2024-05-03T17:04:29.208 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-03T17:04:29.208 DEBUG:teuthology.run:Teuthology command: teuthology --name rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391 --verbose --owner scheduled_rishabh@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} -- /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391/orig.config.yaml 2024-05-03T17:04:29.265 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-03T17:04:29.366 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391 branch: wip-rishabh-testing-20240501.193033 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} email: null first_in_suite: false job_id: '7687391' kernel: client: sha1: distro flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-rishabh-testing-20240501.193033 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 9a047fe7599a549042d06a3fb5b6813a203eabea subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt install: ceph: flavor: default sha1: 9a047fe7599a549042d06a3fb5b6813a203eabea kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240501.193033 sha1: 9a047fe7599a549042d06a3fb5b6813a203eabea owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1698 sha1: 9a047fe7599a549042d06a3fb5b6813a203eabea sleep_before_teardown: 0 subset: 434/512 suite: fs suite_branch: wip-rishabh-testing-20240501.193033 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9a047fe7599a549042d06a3fb5b6813a203eabea/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9a047fe7599a549042d06a3fb5b6813a203eabea syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi067.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEghdIU/zEWluAfDO+6rnEN0WLyvfO5LaMhof4BZIY/yquAmAhE0/8SNhwgf3fyCuI2mEOlh5LoX95aGqe/YkSk= smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAF1kzR9TV75MPp4yQ3LcDRoCrShYqaoDFQ2qELokT0YodJ0JFitAOHRwSMtv4yiFyuR44T4XAtpVvS5Mqr6DQY= smithi082.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBHV1K9B4ZJO3ai6nVSN2vyZxcXSDIAQmpRbv8PH+eQYACgkt58P1vYnFQVfylu2nF4WQuzmAas9/WFPXLkdRCA= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - quiescer: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-03_13:56:19 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-03T17:04:29.366 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9a047fe7599a549042d06a3fb5b6813a203eabea/qa; will attempt to use it 2024-05-03T17:04:29.367 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9a047fe7599a549042d06a3fb5b6813a203eabea/qa/tasks 2024-05-03T17:04:29.367 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-03T17:04:29.368 INFO:teuthology.task.internal:Checking packages... 2024-05-03T17:04:29.393 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9a047fe7599a549042d06a3fb5b6813a203eabea' 2024-05-03T17:04:29.394 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-03T17:04:29.394 INFO:teuthology.packaging:ref: None 2024-05-03T17:04:29.394 INFO:teuthology.packaging:tag: None 2024-05-03T17:04:29.394 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240501.193033 2024-05-03T17:04:29.394 INFO:teuthology.packaging:sha1: 9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T17:04:29.395 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-rishabh-testing-20240501.193033 2024-05-03T17:04:29.612 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3450.g9a047fe7 2024-05-03T17:04:29.614 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-03T17:04:29.620 INFO:teuthology.task.internal:no buildpackages task found 2024-05-03T17:04:29.621 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-03T17:04:29.656 INFO:teuthology.task.internal:Saving configuration 2024-05-03T17:04:29.676 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-03T17:04:29.681 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-03T17:04:29.708 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 16:59:12.192530', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEghdIU/zEWluAfDO+6rnEN0WLyvfO5LaMhof4BZIY/yquAmAhE0/8SNhwgf3fyCuI2mEOlh5LoX95aGqe/YkSk='} 2024-05-03T17:04:29.731 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 16:59:12.190566', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAF1kzR9TV75MPp4yQ3LcDRoCrShYqaoDFQ2qELokT0YodJ0JFitAOHRwSMtv4yiFyuR44T4XAtpVvS5Mqr6DQY='} 2024-05-03T17:04:29.751 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 16:59:12.194261', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBHV1K9B4ZJO3ai6nVSN2vyZxcXSDIAQmpRbv8PH+eQYACgkt58P1vYnFQVfylu2nF4WQuzmAas9/WFPXLkdRCA='} 2024-05-03T17:04:29.751 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-03T17:04:29.757 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-03T17:04:29.757 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-03T17:04:29.757 INFO:teuthology.task.internal:roles: ubuntu@smithi082.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-03T17:04:29.757 INFO:teuthology.run_tasks:Running task console_log... 2024-05-03T17:04:29.850 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f33591a1160>, signals=[15]) 2024-05-03T17:04:29.851 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-03T17:04:29.857 INFO:teuthology.task.internal:Opening connections... 2024-05-03T17:04:29.857 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2024-05-03T17:04:29.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T17:04:29.936 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-05-03T17:04:29.937 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T17:04:30.014 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi082.front.sepia.ceph.com 2024-05-03T17:04:30.015 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T17:04:30.089 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-03T17:04:30.096 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2024-05-03T17:04:30.116 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2024-05-03T17:04:30.116 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2024-05-03T17:04:30.174 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="CentOS Stream" 2024-05-03T17:04:30.175 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="9" 2024-05-03T17:04:30.175 INFO:teuthology.orchestra.run.smithi067.stdout:ID="centos" 2024-05-03T17:04:30.175 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="rhel fedora" 2024-05-03T17:04:30.175 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="9" 2024-05-03T17:04:30.175 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el9" 2024-05-03T17:04:30.176 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-03T17:04:30.176 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2024-05-03T17:04:30.176 INFO:teuthology.orchestra.run.smithi067.stdout:LOGO="fedora-logo-icon" 2024-05-03T17:04:30.176 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-03T17:04:30.176 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://centos.org/" 2024-05-03T17:04:30.176 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-03T17:04:30.176 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-03T17:04:30.176 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-03T17:04:30.177 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2024-05-03T17:04:30.200 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-05-03T17:04:30.218 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-05-03T17:04:30.218 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-05-03T17:04:30.275 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-05-03T17:04:30.275 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="9" 2024-05-03T17:04:30.276 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-05-03T17:04:30.276 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-05-03T17:04:30.276 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="9" 2024-05-03T17:04:30.276 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el9" 2024-05-03T17:04:30.276 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-03T17:04:30.276 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-05-03T17:04:30.276 INFO:teuthology.orchestra.run.smithi070.stdout:LOGO="fedora-logo-icon" 2024-05-03T17:04:30.276 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-03T17:04:30.276 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-05-03T17:04:30.276 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-03T17:04:30.276 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-03T17:04:30.276 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-03T17:04:30.277 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-05-03T17:04:30.299 DEBUG:teuthology.orchestra.run.smithi082:> uname -m 2024-05-03T17:04:30.317 INFO:teuthology.orchestra.run.smithi082.stdout:x86_64 2024-05-03T17:04:30.317 DEBUG:teuthology.orchestra.run.smithi082:> cat /etc/os-release 2024-05-03T17:04:30.374 INFO:teuthology.orchestra.run.smithi082.stdout:NAME="CentOS Stream" 2024-05-03T17:04:30.374 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION="9" 2024-05-03T17:04:30.374 INFO:teuthology.orchestra.run.smithi082.stdout:ID="centos" 2024-05-03T17:04:30.374 INFO:teuthology.orchestra.run.smithi082.stdout:ID_LIKE="rhel fedora" 2024-05-03T17:04:30.374 INFO:teuthology.orchestra.run.smithi082.stdout:VERSION_ID="9" 2024-05-03T17:04:30.374 INFO:teuthology.orchestra.run.smithi082.stdout:PLATFORM_ID="platform:el9" 2024-05-03T17:04:30.374 INFO:teuthology.orchestra.run.smithi082.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-03T17:04:30.375 INFO:teuthology.orchestra.run.smithi082.stdout:ANSI_COLOR="0;31" 2024-05-03T17:04:30.375 INFO:teuthology.orchestra.run.smithi082.stdout:LOGO="fedora-logo-icon" 2024-05-03T17:04:30.375 INFO:teuthology.orchestra.run.smithi082.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-03T17:04:30.375 INFO:teuthology.orchestra.run.smithi082.stdout:HOME_URL="https://centos.org/" 2024-05-03T17:04:30.375 INFO:teuthology.orchestra.run.smithi082.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-03T17:04:30.375 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-03T17:04:30.375 INFO:teuthology.orchestra.run.smithi082.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-03T17:04:30.375 INFO:teuthology.lock.ops:Updating smithi082.front.sepia.ceph.com on lock server 2024-05-03T17:04:30.399 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-03T17:04:30.409 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-03T17:04:30.415 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-03T17:04:30.415 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2024-05-03T17:04:30.418 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-05-03T17:04:30.420 DEBUG:teuthology.orchestra.run.smithi082:> test '!' -e /home/ubuntu/cephtest 2024-05-03T17:04:30.436 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-03T17:04:30.452 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-03T17:04:30.452 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2024-05-03T17:04:30.476 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-05-03T17:04:30.478 DEBUG:teuthology.orchestra.run.smithi082:> test -z $(ls -A /var/lib/ceph) 2024-05-03T17:04:30.505 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-03T17:04:30.585 INFO:teuthology.run_tasks:Running task kernel... 2024-05-03T17:04:30.600 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-05-03T17:04:30.600 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-05-03T17:04:30.600 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-05-03T17:04:30.600 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-03T17:04:30.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:04:30.618 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2024-05-03T17:04:30.676 INFO:teuthology.orchestra.run.smithi067.stdout:5.14.0-437.el9.x86_64 2024-05-03T17:04:30.676 INFO:teuthology.task.kernel:Running kernel on smithi067: 5.14.0-437.el9.x86_64 2024-05-03T17:04:30.676 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2024-05-03T17:04:32.759 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.0 MB 00:00 2024-05-03T17:04:34.668 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 24 MB/s | 19 MB 00:00 2024-05-03T17:04:38.894 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-05-03T17:04:40.303 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 16 kB 00:00 2024-05-03T17:04:41.805 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-05-03T17:04:47.005 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-05-03T17:04:49.094 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-03T17:04:49.095 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-03T17:04:49.186 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-03T17:04:49.187 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2024-05-03T17:04:49.187 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-05-03T17:04:49.312 DEBUG:teuthology.orchestra.run.smithi067:> echo no | sudo yum reinstall kernel || true 2024-05-03T17:04:49.833 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:03 ago on Fri 03 May 2024 05:04:46 PM UTC. 2024-05-03T17:04:50.073 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-03T17:04:50.081 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-03T17:04:50.081 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-05-03T17:04:50.081 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-03T17:04:50.082 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalling: 2024-05-03T17:04:50.082 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-03T17:04:50.082 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:04:50.082 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-05-03T17:04:50.082 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-03T17:04:50.082 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:04:50.082 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 255 k 2024-05-03T17:04:50.082 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 0 2024-05-03T17:04:50.082 INFO:teuthology.orchestra.run.smithi067.stderr:Operation aborted. 2024-05-03T17:04:50.143 INFO:teuthology.orchestra.run.smithi067.stdout:Is this ok [y/N]: 2024-05-03T17:04:50.144 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum reinstall -y kernel || true 2024-05-03T17:04:50.661 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:04 ago on Fri 03 May 2024 05:04:46 PM UTC. 2024-05-03T17:04:50.781 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-03T17:04:50.781 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-03T17:04:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: Package Architecture Version Repository Size 2024-05-03T17:04:50.782 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-03T17:04:50.782 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalling: 2024-05-03T17:04:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: kernel x86_64 5.14.0-437.el9 baseos 255 k 2024-05-03T17:04:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:04:50.782 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-05-03T17:04:50.782 INFO:teuthology.orchestra.run.smithi067.stdout:================================================================================ 2024-05-03T17:04:50.782 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:04:50.782 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 255 k 2024-05-03T17:04:50.782 INFO:teuthology.orchestra.run.smithi067.stdout:Installed size: 0 2024-05-03T17:04:50.782 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-05-03T17:04:50.990 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-437.el9.x86_64.rpm 1.8 MB/s | 255 kB 00:00 2024-05-03T17:04:50.991 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-05-03T17:04:50.991 INFO:teuthology.orchestra.run.smithi067.stdout:Total 1.2 MB/s | 255 kB 00:00 2024-05-03T17:04:51.013 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-05-03T17:04:51.203 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-05-03T17:04:51.204 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-05-03T17:04:51.209 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-05-03T17:04:51.209 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-05-03T17:04:51.356 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-05-03T17:04:51.416 INFO:teuthology.orchestra.run.smithi067.stdout: Reinstalling : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-03T17:04:51.416 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-03T17:04:52.145 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: kernel-5.14.0-437.el9.x86_64 2/2 2024-05-03T17:04:52.145 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 1/2 2024-05-03T17:04:52.888 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : kernel-5.14.0-437.el9.x86_64 2/2 2024-05-03T17:04:52.888 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:04:52.889 INFO:teuthology.orchestra.run.smithi067.stdout:Reinstalled: 2024-05-03T17:04:52.889 INFO:teuthology.orchestra.run.smithi067.stdout: kernel-5.14.0-437.el9.x86_64 2024-05-03T17:04:52.889 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:04:52.889 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-05-03T17:04:53.047 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2024-05-03T17:04:53.095 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-5.14.0-437.el9.x86_64 2024-05-03T17:04:53.096 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-437.el9.x86_64 2024-05-03T17:04:53.096 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2024-05-03T17:04:53.096 DEBUG:teuthology.parallel:result is None 2024-05-03T17:04:53.096 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-03T17:04:53.103 INFO:teuthology.task.internal:Creating test directory... 2024-05-03T17:04:53.104 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-03T17:04:53.106 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-03T17:04:53.110 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-03T17:04:53.127 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-03T17:04:53.135 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-03T17:04:53.142 INFO:teuthology.task.internal:Creating archive directory... 2024-05-03T17:04:53.142 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-03T17:04:53.161 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-03T17:04:53.167 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-03T17:04:53.282 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-03T17:04:53.289 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-03T17:04:53.289 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-03T17:04:53.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:04:53.305 DEBUG:teuthology.orchestra.run.smithi082:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-03T17:04:53.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:04:53.322 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-03T17:04:53.325 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-03T17:04:53.348 DEBUG:teuthology.orchestra.run.smithi082:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-03T17:04:53.351 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T17:04:53.368 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T17:04:53.391 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T17:04:53.403 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T17:04:53.422 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T17:04:53.435 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-03T17:04:53.436 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-03T17:04:53.444 INFO:teuthology.task.internal:Configuring sudo... 2024-05-03T17:04:53.444 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-03T17:04:53.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-03T17:04:53.449 DEBUG:teuthology.orchestra.run.smithi082:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-03T17:04:53.492 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-03T17:04:53.501 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-03T17:04:53.501 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-03T17:04:53.512 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-03T17:04:53.516 DEBUG:teuthology.orchestra.run.smithi082:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-03T17:04:53.547 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T17:04:53.605 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T17:04:53.652 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T17:04:53.733 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T17:04:53.774 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:04:53.774 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-03T17:04:53.838 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T17:04:53.883 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T17:04:53.932 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T17:04:54.014 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T17:04:54.057 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-05-03T17:04:54.057 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-03T17:04:54.122 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T17:04:54.166 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-03T17:04:54.216 DEBUG:teuthology.orchestra.run.smithi082:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T17:04:54.300 DEBUG:teuthology.orchestra.run.smithi082:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-03T17:04:54.340 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T17:04:54.341 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-03T17:04:54.406 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2024-05-03T17:04:54.409 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-05-03T17:04:54.411 DEBUG:teuthology.orchestra.run.smithi082:> sudo service rsyslog restart 2024-05-03T17:04:54.441 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T17:04:54.444 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T17:04:54.485 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T17:04:54.845 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-03T17:04:54.852 INFO:teuthology.task.internal:Starting timer... 2024-05-03T17:04:54.853 INFO:teuthology.run_tasks:Running task pcp... 2024-05-03T17:04:54.863 INFO:teuthology.run_tasks:Running task selinux... 2024-05-03T17:04:54.871 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-03T17:04:54.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2024-05-03T17:04:54.955 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: 2024-05-03T17:04:54.958 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-05-03T17:04:55.018 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-05-03T17:04:55.020 DEBUG:teuthology.orchestra.run.smithi082:> sudo service auditd rotate 2024-05-03T17:04:55.083 INFO:teuthology.orchestra.run.smithi082.stdout:Rotating logs: 2024-05-03T17:04:55.085 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-03T17:04:55.085 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2024-05-03T17:04:55.107 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2024-05-03T17:04:55.107 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-05-03T17:04:55.133 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-05-03T17:04:55.133 DEBUG:teuthology.orchestra.run.smithi082:> /usr/sbin/getenforce 2024-05-03T17:04:55.162 INFO:teuthology.orchestra.run.smithi082.stdout:Permissive 2024-05-03T17:04:55.162 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi070.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi082.front.sepia.ceph.com': 'permissive'} 2024-05-03T17:04:55.162 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-03T17:04:55.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:04:55.188 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-03T17:04:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:04:55.215 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-03T17:04:55.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:04:55.239 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-03T17:04:55.239 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2024-05-03T17:04:55.264 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-05-03T17:04:55.291 DEBUG:teuthology.orchestra.run.smithi082:> sudo /usr/sbin/setenforce permissive 2024-05-03T17:04:55.325 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-03T17:04:55.339 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-03T17:04:55.358 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-03T17:04:55.359 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi070.front.sepia.ceph.com,smithi082.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-03T17:09:05.350 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi070.front.sepia.ceph.com'), Remote(name='ubuntu@smithi082.front.sepia.ceph.com')] 2024-05-03T17:09:05.351 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-03T17:09:05.352 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T17:09:05.429 DEBUG:teuthology.orchestra.run.smithi067:> true 2024-05-03T17:09:05.513 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2024-05-03T17:09:05.514 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-05-03T17:09:05.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T17:09:05.589 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-05-03T17:09:05.625 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-05-03T17:09:05.625 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-05-03T17:09:05.626 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi082.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-03T17:09:05.703 DEBUG:teuthology.orchestra.run.smithi082:> true 2024-05-03T17:09:05.778 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi082.front.sepia.ceph.com' 2024-05-03T17:09:05.778 INFO:teuthology.run_tasks:Running task clock... 2024-05-03T17:09:05.789 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-03T17:09:05.789 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-03T17:09:05.790 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T17:09:05.792 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-03T17:09:05.792 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T17:09:05.794 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-03T17:09:05.795 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T17:09:05.821 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-03T17:09:05.821 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-03T17:09:05.834 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-03T17:09:05.835 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-03T17:09:05.847 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-03T17:09:05.857 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2024-05-03T17:09:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2024-05-03T17:09:05.861 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-03T17:09:05.867 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2024-05-03T17:09:05.871 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2024-05-03T17:09:05.879 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-03T17:09:05.883 INFO:teuthology.orchestra.run.smithi082.stderr:sudo: ntpd: command not found 2024-05-03T17:09:05.885 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-03T17:09:05.892 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-03T17:09:05.894 INFO:teuthology.orchestra.run.smithi082.stdout:506 Cannot talk to daemon 2024-05-03T17:09:05.899 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-03T17:09:05.907 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-03T17:09:05.919 INFO:teuthology.orchestra.run.smithi082.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-03T17:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-05-03T17:09:05.951 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T17:09:05.951 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-05-03T17:09:05.951 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:09:05.951 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:09:05.951 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:09:05.951 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:09:05.954 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-05-03T17:09:05.958 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T17:09:05.958 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-05-03T17:09:05.958 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:09:05.958 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:09:05.958 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:09:05.958 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:09:05.975 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-05-03T17:09:05.979 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T17:09:05.979 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-05-03T17:09:05.980 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:09:05.980 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:09:05.980 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:09:05.980 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:09:05.980 INFO:teuthology.run_tasks:Running task install... 2024-05-03T17:09:05.988 DEBUG:teuthology.task.install:project ceph 2024-05-03T17:09:05.988 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea'}} 2024-05-03T17:09:05.988 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea'} 2024-05-03T17:09:05.988 INFO:teuthology.task.install:Using flavor: default 2024-05-03T17:09:05.994 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-03T17:09:05.994 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-03T17:09:05.995 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea', 'tag': None, 'wait_for_package': False} 2024-05-03T17:09:05.995 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T17:09:05.997 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea', 'tag': None, 'wait_for_package': False} 2024-05-03T17:09:05.997 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T17:09:05.998 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea', 'tag': None, 'wait_for_package': False} 2024-05-03T17:09:05.998 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T17:09:06.153 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/ 2024-05-03T17:09:06.153 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3450.g9a047fe7 2024-05-03T17:09:06.219 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/ 2024-05-03T17:09:06.219 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3450.g9a047fe7 2024-05-03T17:09:06.222 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/ 2024-05-03T17:09:06.223 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3450.g9a047fe7 2024-05-03T17:09:06.266 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-03T17:09:06.267 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:09:06.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-03T17:09:06.293 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-03T17:09:06.293 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9a047fe7599a549042d06a3fb5b6813a203eabea/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-03T17:09:06.338 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-03T17:09:06.338 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T17:09:06.339 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-03T17:09:06.359 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-03T17:09:06.363 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-03T17:09:06.363 DEBUG:teuthology.orchestra.run.smithi082:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9a047fe7599a549042d06a3fb5b6813a203eabea/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-03T17:09:06.393 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-03T17:09:06.393 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-05-03T17:09:06.393 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-03T17:09:06.417 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-03T17:09:06.418 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9a047fe7599a549042d06a3fb5b6813a203eabea/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-03T17:09:06.427 DEBUG:teuthology.orchestra.run.smithi082:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-03T17:09:06.434 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-03T17:09:06.482 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-03T17:09:06.498 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2024-05-03T17:09:06.500 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2024-05-03T17:09:06.501 DEBUG:teuthology.orchestra.run.smithi082:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-03T17:09:06.555 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-03T17:09:06.563 INFO:teuthology.orchestra.run.smithi082.stdout:check_obsoletes = 1 2024-05-03T17:09:06.565 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum clean all 2024-05-03T17:09:06.618 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2024-05-03T17:09:06.619 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2024-05-03T17:09:06.735 INFO:teuthology.orchestra.run.smithi067.stdout:50 files removed 2024-05-03T17:09:06.762 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-03T17:09:06.791 INFO:teuthology.orchestra.run.smithi082.stdout:50 files removed 2024-05-03T17:09:06.818 DEBUG:teuthology.orchestra.run.smithi082:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-03T17:09:06.844 INFO:teuthology.orchestra.run.smithi070.stdout:50 files removed 2024-05-03T17:09:06.871 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-03T17:09:07.510 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 176 kB/s | 84 kB 00:00 2024-05-03T17:09:07.532 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 208 kB/s | 84 kB 00:00 2024-05-03T17:09:07.762 INFO:teuthology.orchestra.run.smithi082.stdout:ceph packages for x86_64 122 kB/s | 84 kB 00:00 2024-05-03T17:09:07.894 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 35 kB/s | 12 kB 00:00 2024-05-03T17:09:08.121 INFO:teuthology.orchestra.run.smithi082.stdout:ceph noarch packages 37 kB/s | 12 kB 00:00 2024-05-03T17:09:08.199 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 19 kB/s | 12 kB 00:00 2024-05-03T17:09:08.203 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 7.0 kB/s | 1.9 kB 00:00 2024-05-03T17:09:08.571 INFO:teuthology.orchestra.run.smithi082.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-05-03T17:09:08.791 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 3.4 kB/s | 1.9 kB 00:00 2024-05-03T17:09:09.153 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - BaseOS 8.8 MB/s | 8.0 MB 00:00 2024-05-03T17:09:12.916 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 2.0 MB/s | 8.0 MB 00:04 2024-05-03T17:09:13.412 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 19 MB 00:03 2024-05-03T17:09:13.579 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - BaseOS 1.6 MB/s | 8.0 MB 00:04 2024-05-03T17:09:14.995 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:00 2024-05-03T17:09:17.038 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 19 MB 00:02 2024-05-03T17:09:18.388 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - CRB 4.8 MB/s | 6.2 MB 00:01 2024-05-03T17:09:19.151 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-05-03T17:09:20.039 INFO:teuthology.orchestra.run.smithi067.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-05-03T17:09:21.161 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 16 kB 00:00 2024-05-03T17:09:21.957 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-03T17:09:22.070 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-05-03T17:09:24.121 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.2 MB 00:03 2024-05-03T17:09:26.039 INFO:teuthology.orchestra.run.smithi082.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-05-03T17:09:27.211 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-05-03T17:09:27.233 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-03T17:09:27.257 INFO:teuthology.orchestra.run.smithi082.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-05-03T17:09:29.371 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-03T17:09:29.371 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-03T17:09:29.382 INFO:teuthology.orchestra.run.smithi070.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-03T17:09:29.383 INFO:teuthology.orchestra.run.smithi070.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-03T17:09:29.384 INFO:teuthology.orchestra.run.smithi070.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-03T17:09:29.384 INFO:teuthology.orchestra.run.smithi070.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-03T17:09:29.386 INFO:teuthology.orchestra.run.smithi070.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-03T17:09:29.388 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-03T17:09:29.389 INFO:teuthology.orchestra.run.smithi070.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-03T17:09:29.426 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-03T17:09:29.426 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-03T17:09:29.437 INFO:teuthology.orchestra.run.smithi067.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-03T17:09:29.438 INFO:teuthology.orchestra.run.smithi067.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-03T17:09:29.439 INFO:teuthology.orchestra.run.smithi067.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-03T17:09:29.440 INFO:teuthology.orchestra.run.smithi067.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-03T17:09:29.442 INFO:teuthology.orchestra.run.smithi067.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-03T17:09:29.444 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-03T17:09:29.445 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-03T17:09:29.450 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-05-03T17:09:29.470 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================= 2024-05-03T17:09:29.470 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-05-03T17:09:29.470 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================= 2024-05-03T17:09:29.470 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-05-03T17:09:29.470 INFO:teuthology.orchestra.run.smithi070.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-03T17:09:29.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 6.6 k 2024-05-03T17:09:29.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 5.4 M 2024-05-03T17:09:29.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 891 k 2024-05-03T17:09:29.470 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 145 k 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 1.5 M 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 142 k 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 3.5 M 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 7.4 M 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 49 k 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 11 M 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 49 M 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 273 k 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 763 k 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 225 k 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-shell x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 27 k 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 18 k 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-03T17:09:29.471 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 32 k 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 747 k 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 127 k 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 161 k 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 320 k 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 298 k 2024-05-03T17:09:29.472 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 100 k 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 85 k 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.1 M 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 170 k 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.4 M 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.1 M 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-03T17:09:29.473 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 21 M 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 34 k 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 2.4 M 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 274 k 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 4.7 M 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 17 M 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 16 k 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 25 k 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-03T17:09:29.474 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 163 k 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 500 k 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 5.4 M 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-03T17:09:29.475 INFO:teuthology.orchestra.run.smithi070.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-03T17:09:29.476 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 45 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 135 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-03T17:09:29.477 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-03T17:09:29.478 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-03T17:09:29.479 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-03T17:09:29.480 INFO:teuthology.orchestra.run.smithi070.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout:Installing weak dependencies: 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================= 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout:Install 148 Packages 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 4 Packages 2024-05-03T17:09:29.481 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:09:29.482 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 222 M 2024-05-03T17:09:29.482 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-05-03T17:09:29.509 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2024-05-03T17:09:29.518 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-05-03T17:09:29.518 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 6.6 k 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 5.4 M 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 891 k 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 145 k 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 1.5 M 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 142 k 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 3.5 M 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 7.4 M 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 49 k 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 11 M 2024-05-03T17:09:29.519 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 49 M 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 273 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 763 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 225 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 27 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 18 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 32 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 747 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 127 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-03T17:09:29.520 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 161 k 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 320 k 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 298 k 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 100 k 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 85 k 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.1 M 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 170 k 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.4 M 2024-05-03T17:09:29.521 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.1 M 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 21 M 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 34 k 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 2.4 M 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 274 k 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 4.7 M 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 17 M 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 16 k 2024-05-03T17:09:29.522 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 25 k 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 163 k 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-03T17:09:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 500 k 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 5.4 M 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-03T17:09:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 45 k 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 135 k 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-03T17:09:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-03T17:09:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-03T17:09:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-03T17:09:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout:Installing weak dependencies: 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout:======================================================================================= 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout:Install 148 Packages 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 4 Packages 2024-05-03T17:09:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:09:29.530 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 222 M 2024-05-03T17:09:29.530 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2024-05-03T17:09:30.476 INFO:teuthology.orchestra.run.smithi067.stdout:(1/152): ceph-19.0.0-3450.g9a047fe7.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-05-03T17:09:30.487 INFO:teuthology.orchestra.run.smithi070.stdout:(1/152): ceph-19.0.0-3450.g9a047fe7.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-05-03T17:09:30.854 INFO:teuthology.orchestra.run.smithi070.stdout:(2/152): ceph-base-19.0.0-3450.g9a047fe7.el9.x8 7.8 MB/s | 5.4 MB 00:00 2024-05-03T17:09:30.913 INFO:teuthology.orchestra.run.smithi070.stdout:(3/152): ceph-immutable-object-cache-19.0.0-345 2.5 MB/s | 145 kB 00:00 2024-05-03T17:09:31.171 INFO:teuthology.orchestra.run.smithi070.stdout:(4/152): ceph-mds-19.0.0-3450.g9a047fe7.el9.x86 9.1 MB/s | 2.4 MB 00:00 2024-05-03T17:09:31.363 INFO:teuthology.orchestra.run.smithi070.stdout:(5/152): ceph-mgr-19.0.0-3450.g9a047fe7.el9.x86 7.9 MB/s | 1.5 MB 00:00 2024-05-03T17:09:31.567 INFO:teuthology.orchestra.run.smithi067.stdout:(2/152): ceph-fuse-19.0.0-3450.g9a047fe7.el9.x8 816 kB/s | 891 kB 00:01 2024-05-03T17:09:31.839 INFO:teuthology.orchestra.run.smithi070.stdout:(6/152): ceph-mon-19.0.0-3450.g9a047fe7.el9.x86 9.9 MB/s | 4.7 MB 00:00 2024-05-03T17:09:31.872 INFO:teuthology.orchestra.run.smithi070.stdout:(7/152): ceph-fuse-19.0.0-3450.g9a047fe7.el9.x8 644 kB/s | 891 kB 00:01 2024-05-03T17:09:31.968 INFO:teuthology.orchestra.run.smithi067.stdout:(3/152): ceph-common-19.0.0-3450.g9a047fe7.el9. 12 MB/s | 21 MB 00:01 2024-05-03T17:09:31.993 INFO:teuthology.orchestra.run.smithi067.stdout:(4/152): ceph-immutable-object-cache-19.0.0-345 340 kB/s | 145 kB 00:00 2024-05-03T17:09:32.348 INFO:teuthology.orchestra.run.smithi082.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-05-03T17:09:32.460 INFO:teuthology.orchestra.run.smithi067.stdout:(5/152): ceph-mds-19.0.0-3450.g9a047fe7.el9.x86 4.8 MB/s | 2.4 MB 00:00 2024-05-03T17:09:32.506 INFO:teuthology.orchestra.run.smithi070.stdout:(8/152): ceph-radosgw-19.0.0-3450.g9a047fe7.el9 17 MB/s | 11 MB 00:00 2024-05-03T17:09:32.564 INFO:teuthology.orchestra.run.smithi070.stdout:(9/152): ceph-selinux-19.0.0-3450.g9a047fe7.el9 434 kB/s | 25 kB 00:00 2024-05-03T17:09:32.894 INFO:teuthology.orchestra.run.smithi067.stdout:(6/152): ceph-mon-19.0.0-3450.g9a047fe7.el9.x86 11 MB/s | 4.7 MB 00:00 2024-05-03T17:09:33.861 INFO:teuthology.orchestra.run.smithi067.stdout:(7/152): ceph-mgr-19.0.0-3450.g9a047fe7.el9.x86 829 kB/s | 1.5 MB 00:01 2024-05-03T17:09:34.483 INFO:teuthology.orchestra.run.smithi082.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-03T17:09:34.484 INFO:teuthology.orchestra.run.smithi082.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-03T17:09:34.495 INFO:teuthology.orchestra.run.smithi082.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-03T17:09:34.495 INFO:teuthology.orchestra.run.smithi082.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-03T17:09:34.496 INFO:teuthology.orchestra.run.smithi082.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-03T17:09:34.497 INFO:teuthology.orchestra.run.smithi082.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-03T17:09:34.499 INFO:teuthology.orchestra.run.smithi082.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-03T17:09:34.501 INFO:teuthology.orchestra.run.smithi082.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-03T17:09:34.502 INFO:teuthology.orchestra.run.smithi082.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-03T17:09:34.563 INFO:teuthology.orchestra.run.smithi082.stdout:Dependencies resolved. 2024-05-03T17:09:34.581 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-05-03T17:09:34.581 INFO:teuthology.orchestra.run.smithi082.stdout: Package Arch Version Repository Size 2024-05-03T17:09:34.581 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout:Installing: 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 6.6 k 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 5.4 M 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 891 k 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 145 k 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 1.5 M 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 142 k 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 3.5 M 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 7.4 M 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 49 k 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 11 M 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 49 M 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 273 k 2024-05-03T17:09:34.582 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 763 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-mirror x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 225 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-shell x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 27 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-top noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 18 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 32 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 747 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 127 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-03T17:09:34.583 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 161 k 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 320 k 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 298 k 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 100 k 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 85 k 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.1 M 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 170 k 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrading: 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: librados2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.4 M 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 3.1 M 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout:Installing dependencies: 2024-05-03T17:09:34.584 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 21 M 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 34 k 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 2.4 M 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 274 k 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 4.7 M 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 17 M 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3450.g9a047fe7.el9 ceph-noarch 16 k 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 25 k 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-03T17:09:34.585 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 163 k 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 500 k 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2 x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 5.4 M 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-03T17:09:34.586 INFO:teuthology.orchestra.run.smithi082.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 45 k 2024-05-03T17:09:34.587 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common x86_64 2:19.0.0-3450.g9a047fe7.el9 ceph 135 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-03T17:09:34.588 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-03T17:09:34.589 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-03T17:09:34.590 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-03T17:09:34.591 INFO:teuthology.orchestra.run.smithi082.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-03T17:09:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-03T17:09:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-03T17:09:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-03T17:09:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-03T17:09:34.592 INFO:teuthology.orchestra.run.smithi082.stdout:Installing weak dependencies: 2024-05-03T17:09:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-03T17:09:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:09:34.592 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction Summary 2024-05-03T17:09:34.592 INFO:teuthology.orchestra.run.smithi082.stdout:======================================================================================= 2024-05-03T17:09:34.592 INFO:teuthology.orchestra.run.smithi082.stdout:Install 148 Packages 2024-05-03T17:09:34.592 INFO:teuthology.orchestra.run.smithi082.stdout:Upgrade 4 Packages 2024-05-03T17:09:34.592 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:09:34.593 INFO:teuthology.orchestra.run.smithi082.stdout:Total download size: 222 M 2024-05-03T17:09:34.593 INFO:teuthology.orchestra.run.smithi082.stdout:Downloading Packages: 2024-05-03T17:09:34.661 INFO:teuthology.orchestra.run.smithi067.stdout:(8/152): ceph-osd-19.0.0-3450.g9a047fe7.el9.x86 9.5 MB/s | 17 MB 00:01 2024-05-03T17:09:34.736 INFO:teuthology.orchestra.run.smithi067.stdout:(9/152): ceph-selinux-19.0.0-3450.g9a047fe7.el9 337 kB/s | 25 kB 00:00 2024-05-03T17:09:34.906 INFO:teuthology.orchestra.run.smithi070.stdout:(10/152): ceph-common-19.0.0-3450.g9a047fe7.el9 4.5 MB/s | 21 MB 00:04 2024-05-03T17:09:34.998 INFO:teuthology.orchestra.run.smithi070.stdout:(11/152): cephfs-mirror-19.0.0-3450.g9a047fe7.e 2.4 MB/s | 225 kB 00:00 2024-05-03T17:09:35.073 INFO:teuthology.orchestra.run.smithi070.stdout:(12/152): cephfs-shell-19.0.0-3450.g9a047fe7.el 360 kB/s | 27 kB 00:00 2024-05-03T17:09:35.156 INFO:teuthology.orchestra.run.smithi070.stdout:(13/152): libcephfs-devel-19.0.0-3450.g9a047fe7 380 kB/s | 32 kB 00:00 2024-05-03T17:09:35.324 INFO:teuthology.orchestra.run.smithi070.stdout:(14/152): libcephfs2-19.0.0-3450.g9a047fe7.el9. 4.4 MB/s | 747 kB 00:00 2024-05-03T17:09:35.415 INFO:teuthology.orchestra.run.smithi070.stdout:(15/152): libcephsqlite-19.0.0-3450.g9a047fe7.e 1.7 MB/s | 163 kB 00:00 2024-05-03T17:09:35.466 INFO:teuthology.orchestra.run.smithi082.stdout:(1/152): ceph-19.0.0-3450.g9a047fe7.el9.x86_64. 16 kB/s | 6.6 kB 00:00 2024-05-03T17:09:35.507 INFO:teuthology.orchestra.run.smithi070.stdout:(16/152): librados-devel-19.0.0-3450.g9a047fe7. 1.4 MB/s | 127 kB 00:00 2024-05-03T17:09:35.608 INFO:teuthology.orchestra.run.smithi070.stdout:(17/152): libradosstriper1-19.0.0-3450.g9a047fe 4.9 MB/s | 500 kB 00:00 2024-05-03T17:09:36.450 INFO:teuthology.orchestra.run.smithi070.stdout:(18/152): librgw2-19.0.0-3450.g9a047fe7.el9.x86 6.4 MB/s | 5.4 MB 00:00 2024-05-03T17:09:36.533 INFO:teuthology.orchestra.run.smithi070.stdout:(19/152): python3-ceph-argparse-19.0.0-3450.g9a 543 kB/s | 45 kB 00:00 2024-05-03T17:09:36.607 INFO:teuthology.orchestra.run.smithi082.stdout:(2/152): ceph-fuse-19.0.0-3450.g9a047fe7.el9.x8 781 kB/s | 891 kB 00:01 2024-05-03T17:09:36.617 INFO:teuthology.orchestra.run.smithi070.stdout:(20/152): python3-ceph-common-19.0.0-3450.g9a04 1.6 MB/s | 135 kB 00:00 2024-05-03T17:09:36.700 INFO:teuthology.orchestra.run.smithi070.stdout:(21/152): python3-cephfs-19.0.0-3450.g9a047fe7. 1.9 MB/s | 161 kB 00:00 2024-05-03T17:09:36.791 INFO:teuthology.orchestra.run.smithi082.stdout:(3/152): ceph-immutable-object-cache-19.0.0-345 790 kB/s | 145 kB 00:00 2024-05-03T17:09:36.800 INFO:teuthology.orchestra.run.smithi070.stdout:(22/152): python3-rados-19.0.0-3450.g9a047fe7.e 3.1 MB/s | 320 kB 00:00 2024-05-03T17:09:36.901 INFO:teuthology.orchestra.run.smithi070.stdout:(23/152): python3-rbd-19.0.0-3450.g9a047fe7.el9 2.9 MB/s | 298 kB 00:00 2024-05-03T17:09:36.984 INFO:teuthology.orchestra.run.smithi070.stdout:(24/152): python3-rgw-19.0.0-3450.g9a047fe7.el9 1.2 MB/s | 100 kB 00:00 2024-05-03T17:09:37.033 INFO:teuthology.orchestra.run.smithi082.stdout:(4/152): ceph-base-19.0.0-3450.g9a047fe7.el9.x8 2.7 MB/s | 5.4 MB 00:01 2024-05-03T17:09:37.060 INFO:teuthology.orchestra.run.smithi070.stdout:(25/152): rbd-fuse-19.0.0-3450.g9a047fe7.el9.x8 1.1 MB/s | 85 kB 00:00 2024-05-03T17:09:37.358 INFO:teuthology.orchestra.run.smithi082.stdout:(5/152): ceph-mgr-19.0.0-3450.g9a047fe7.el9.x86 4.7 MB/s | 1.5 MB 00:00 2024-05-03T17:09:37.652 INFO:teuthology.orchestra.run.smithi070.stdout:(26/152): rbd-mirror-19.0.0-3450.g9a047fe7.el9. 5.2 MB/s | 3.1 MB 00:00 2024-05-03T17:09:37.752 INFO:teuthology.orchestra.run.smithi070.stdout:(27/152): rbd-nbd-19.0.0-3450.g9a047fe7.el9.x86 1.7 MB/s | 170 kB 00:00 2024-05-03T17:09:37.844 INFO:teuthology.orchestra.run.smithi070.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3450.g 373 kB/s | 34 kB 00:00 2024-05-03T17:09:37.944 INFO:teuthology.orchestra.run.smithi070.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3450.g9a047fe 1.4 MB/s | 142 kB 00:00 2024-05-03T17:09:37.992 INFO:teuthology.orchestra.run.smithi082.stdout:(6/152): ceph-mon-19.0.0-3450.g9a047fe7.el9.x86 7.4 MB/s | 4.7 MB 00:00 2024-05-03T17:09:38.134 INFO:teuthology.orchestra.run.smithi082.stdout:(7/152): ceph-mds-19.0.0-3450.g9a047fe7.el9.x86 1.8 MB/s | 2.4 MB 00:01 2024-05-03T17:09:38.937 INFO:teuthology.orchestra.run.smithi070.stdout:(30/152): ceph-test-19.0.0-3450.g9a047fe7.el9.x 7.7 MB/s | 49 MB 00:06 2024-05-03T17:09:38.995 INFO:teuthology.orchestra.run.smithi070.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3450.g9a047 3.4 MB/s | 3.5 MB 00:01 2024-05-03T17:09:39.145 INFO:teuthology.orchestra.run.smithi070.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3450.g9a 1.8 MB/s | 274 kB 00:00 2024-05-03T17:09:39.229 INFO:teuthology.orchestra.run.smithi070.stdout:(33/152): ceph-mgr-rook-19.0.0-3450.g9a047fe7.e 594 kB/s | 49 kB 00:00 2024-05-03T17:09:39.312 INFO:teuthology.orchestra.run.smithi070.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3450.g9 197 kB/s | 16 kB 00:00 2024-05-03T17:09:39.404 INFO:teuthology.orchestra.run.smithi070.stdout:(35/152): ceph-volume-19.0.0-3450.g9a047fe7.el9 2.9 MB/s | 273 kB 00:00 2024-05-03T17:09:39.655 INFO:teuthology.orchestra.run.smithi070.stdout:(36/152): cephadm-19.0.0-3450.g9a047fe7.el9.noa 3.0 MB/s | 763 kB 00:00 2024-05-03T17:09:39.713 INFO:teuthology.orchestra.run.smithi070.stdout:(37/152): cephfs-top-19.0.0-3450.g9a047fe7.el9. 313 kB/s | 18 kB 00:00 2024-05-03T17:09:39.906 INFO:teuthology.orchestra.run.smithi070.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 419 kB/s | 80 kB 00:00 2024-05-03T17:09:39.908 INFO:teuthology.orchestra.run.smithi082.stdout:(8/152): ceph-osd-19.0.0-3450.g9a047fe7.el9.x86 8.8 MB/s | 17 MB 00:01 2024-05-03T17:09:39.975 INFO:teuthology.orchestra.run.smithi082.stdout:(9/152): ceph-selinux-19.0.0-3450.g9a047fe7.el9 378 kB/s | 25 kB 00:00 2024-05-03T17:09:40.105 INFO:teuthology.orchestra.run.smithi070.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-05-03T17:09:40.161 INFO:teuthology.orchestra.run.smithi067.stdout:(10/152): ceph-base-19.0.0-3450.g9a047fe7.el9.x 552 kB/s | 5.4 MB 00:09 2024-05-03T17:09:40.289 INFO:teuthology.orchestra.run.smithi070.stdout:(40/152): ceph-mgr-diskprediction-local-19.0.0- 5.5 MB/s | 7.4 MB 00:01 2024-05-03T17:09:40.406 INFO:teuthology.orchestra.run.smithi070.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 650 kB/s | 195 kB 00:00 2024-05-03T17:09:40.565 INFO:teuthology.orchestra.run.smithi070.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 921 kB/s | 253 kB 00:00 2024-05-03T17:09:40.665 INFO:teuthology.orchestra.run.smithi070.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-05-03T17:09:40.732 INFO:teuthology.orchestra.run.smithi070.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-05-03T17:09:40.799 INFO:teuthology.orchestra.run.smithi070.stdout:(45/152): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-05-03T17:09:40.820 INFO:teuthology.orchestra.run.smithi067.stdout:(11/152): ceph-test-19.0.0-3450.g9a047fe7.el9.x 8.1 MB/s | 49 MB 00:06 2024-05-03T17:09:40.845 INFO:teuthology.orchestra.run.smithi067.stdout:(12/152): cephfs-mirror-19.0.0-3450.g9a047fe7.e 329 kB/s | 225 kB 00:00 2024-05-03T17:09:40.849 INFO:teuthology.orchestra.run.smithi070.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 2.8 MB/s | 1.2 MB 00:00 2024-05-03T17:09:40.874 INFO:teuthology.orchestra.run.smithi070.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-05-03T17:09:40.878 INFO:teuthology.orchestra.run.smithi067.stdout:(13/152): cephfs-shell-19.0.0-3450.g9a047fe7.el 465 kB/s | 27 kB 00:00 2024-05-03T17:09:40.929 INFO:teuthology.orchestra.run.smithi067.stdout:(14/152): libcephfs-devel-19.0.0-3450.g9a047fe7 379 kB/s | 32 kB 00:00 2024-05-03T17:09:41.037 INFO:teuthology.orchestra.run.smithi067.stdout:(15/152): libcephfs2-19.0.0-3450.g9a047fe7.el9. 4.6 MB/s | 747 kB 00:00 2024-05-03T17:09:41.104 INFO:teuthology.orchestra.run.smithi067.stdout:(16/152): librados-devel-19.0.0-3450.g9a047fe7. 1.9 MB/s | 127 kB 00:00 2024-05-03T17:09:41.204 INFO:teuthology.orchestra.run.smithi067.stdout:(17/152): libcephsqlite-19.0.0-3450.g9a047fe7.e 593 kB/s | 163 kB 00:00 2024-05-03T17:09:41.216 INFO:teuthology.orchestra.run.smithi070.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 2.7 MB/s | 943 kB 00:00 2024-05-03T17:09:41.230 INFO:teuthology.orchestra.run.smithi067.stdout:(18/152): libradosstriper1-19.0.0-3450.g9a047fe 3.9 MB/s | 500 kB 00:00 2024-05-03T17:09:41.275 INFO:teuthology.orchestra.run.smithi070.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-05-03T17:09:41.297 INFO:teuthology.orchestra.run.smithi067.stdout:(19/152): python3-ceph-argparse-19.0.0-3450.g9a 677 kB/s | 45 kB 00:00 2024-05-03T17:09:41.342 INFO:teuthology.orchestra.run.smithi070.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 322 kB/s | 21 kB 00:00 2024-05-03T17:09:41.363 INFO:teuthology.orchestra.run.smithi067.stdout:(20/152): python3-ceph-common-19.0.0-3450.g9a04 2.0 MB/s | 135 kB 00:00 2024-05-03T17:09:41.417 INFO:teuthology.orchestra.run.smithi070.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 4.1 MB/s | 313 kB 00:00 2024-05-03T17:09:41.431 INFO:teuthology.orchestra.run.smithi067.stdout:(21/152): python3-cephfs-19.0.0-3450.g9a047fe7. 2.4 MB/s | 161 kB 00:00 2024-05-03T17:09:41.451 INFO:teuthology.orchestra.run.smithi070.stdout:(52/152): automake-1.16.2-8.el9.noarch.rpm 1.1 MB/s | 689 kB 00:00 2024-05-03T17:09:41.476 INFO:teuthology.orchestra.run.smithi070.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 556 kB/s | 33 kB 00:00 2024-05-03T17:09:41.506 INFO:teuthology.orchestra.run.smithi067.stdout:(22/152): python3-rados-19.0.0-3450.g9a047fe7.e 4.2 MB/s | 320 kB 00:00 2024-05-03T17:09:41.535 INFO:teuthology.orchestra.run.smithi070.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-05-03T17:09:41.556 INFO:teuthology.orchestra.run.smithi067.stdout:(23/152): python3-rbd-19.0.0-3450.g9a047fe7.el9 5.8 MB/s | 298 kB 00:00 2024-05-03T17:09:41.592 INFO:teuthology.orchestra.run.smithi082.stdout:(10/152): ceph-radosgw-19.0.0-3450.g9a047fe7.el 3.1 MB/s | 11 MB 00:03 2024-05-03T17:09:41.593 INFO:teuthology.orchestra.run.smithi070.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 191 kB/s | 11 kB 00:00 2024-05-03T17:09:41.606 INFO:teuthology.orchestra.run.smithi067.stdout:(24/152): python3-rgw-19.0.0-3450.g9a047fe7.el9 2.0 MB/s | 100 kB 00:00 2024-05-03T17:09:41.652 INFO:teuthology.orchestra.run.smithi070.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-05-03T17:09:41.682 INFO:teuthology.orchestra.run.smithi067.stdout:(25/152): rbd-fuse-19.0.0-3450.g9a047fe7.el9.x8 1.1 MB/s | 85 kB 00:00 2024-05-03T17:09:41.711 INFO:teuthology.orchestra.run.smithi070.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 776 kB/s | 45 kB 00:00 2024-05-03T17:09:41.742 INFO:teuthology.orchestra.run.smithi082.stdout:(11/152): cephfs-mirror-19.0.0-3450.g9a047fe7.e 1.5 MB/s | 225 kB 00:00 2024-05-03T17:09:41.778 INFO:teuthology.orchestra.run.smithi070.stdout:(58/152): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-05-03T17:09:41.845 INFO:teuthology.orchestra.run.smithi070.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 582 kB 00:00 2024-05-03T17:09:41.851 INFO:teuthology.orchestra.run.smithi082.stdout:(12/152): cephfs-shell-19.0.0-3450.g9a047fe7.el 250 kB/s | 27 kB 00:00 2024-05-03T17:09:41.903 INFO:teuthology.orchestra.run.smithi070.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 437 kB/s | 25 kB 00:00 2024-05-03T17:09:41.960 INFO:teuthology.orchestra.run.smithi082.stdout:(13/152): libcephfs-devel-19.0.0-3450.g9a047fe7 293 kB/s | 32 kB 00:00 2024-05-03T17:09:41.962 INFO:teuthology.orchestra.run.smithi070.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 812 kB/s | 47 kB 00:00 2024-05-03T17:09:42.029 INFO:teuthology.orchestra.run.smithi070.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 292 kB 00:00 2024-05-03T17:09:42.096 INFO:teuthology.orchestra.run.smithi070.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-05-03T17:09:42.124 INFO:teuthology.orchestra.run.smithi067.stdout:(26/152): rbd-mirror-19.0.0-3450.g9a047fe7.el9. 7.0 MB/s | 3.1 MB 00:00 2024-05-03T17:09:42.163 INFO:teuthology.orchestra.run.smithi070.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 528 kB/s | 35 kB 00:00 2024-05-03T17:09:42.191 INFO:teuthology.orchestra.run.smithi067.stdout:(27/152): rbd-nbd-19.0.0-3450.g9a047fe7.el9.x86 2.5 MB/s | 170 kB 00:00 2024-05-03T17:09:42.258 INFO:teuthology.orchestra.run.smithi067.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3450.g 513 kB/s | 34 kB 00:00 2024-05-03T17:09:42.310 INFO:teuthology.orchestra.run.smithi082.stdout:(14/152): libcephfs2-19.0.0-3450.g9a047fe7.el9. 2.1 MB/s | 747 kB 00:00 2024-05-03T17:09:42.316 INFO:teuthology.orchestra.run.smithi067.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3450.g9a047fe 2.4 MB/s | 142 kB 00:00 2024-05-03T17:09:42.410 INFO:teuthology.orchestra.run.smithi082.stdout:(15/152): libcephsqlite-19.0.0-3450.g9a047fe7.e 1.6 MB/s | 163 kB 00:00 2024-05-03T17:09:42.421 INFO:teuthology.orchestra.run.smithi070.stdout:(65/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 3.1 MB/s | 3.0 MB 00:00 2024-05-03T17:09:42.489 INFO:teuthology.orchestra.run.smithi070.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 15 MB/s | 4.7 MB 00:00 2024-05-03T17:09:42.527 INFO:teuthology.orchestra.run.smithi082.stdout:(16/152): librados-devel-19.0.0-3450.g9a047fe7. 1.1 MB/s | 127 kB 00:00 2024-05-03T17:09:42.614 INFO:teuthology.orchestra.run.smithi070.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-05-03T17:09:42.698 INFO:teuthology.orchestra.run.smithi070.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-05-03T17:09:42.756 INFO:teuthology.orchestra.run.smithi070.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-05-03T17:09:42.811 INFO:teuthology.orchestra.run.smithi082.stdout:(17/152): libradosstriper1-19.0.0-3450.g9a047fe 1.7 MB/s | 500 kB 00:00 2024-05-03T17:09:42.850 INFO:teuthology.orchestra.run.smithi067.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3450.g9a047 6.6 MB/s | 3.5 MB 00:00 2024-05-03T17:09:43.064 INFO:teuthology.orchestra.run.smithi070.stdout:(70/152): postgresql-server-13.14-1.el9.x86_64. 19 MB/s | 5.8 MB 00:00 2024-05-03T17:09:43.165 INFO:teuthology.orchestra.run.smithi070.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 6.0 MB/s | 4.4 MB 00:00 2024-05-03T17:09:43.207 INFO:teuthology.orchestra.run.smithi070.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 7.1 MB/s | 1.0 MB 00:00 2024-05-03T17:09:43.232 INFO:teuthology.orchestra.run.smithi070.stdout:(73/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 87 kB 00:00 2024-05-03T17:09:43.316 INFO:teuthology.orchestra.run.smithi070.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-05-03T17:09:43.391 INFO:teuthology.orchestra.run.smithi070.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 3.2 MB/s | 248 kB 00:00 2024-05-03T17:09:43.458 INFO:teuthology.orchestra.run.smithi070.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 386 kB/s | 26 kB 00:00 2024-05-03T17:09:43.484 INFO:teuthology.orchestra.run.smithi067.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-05-03T17:09:43.486 INFO:teuthology.orchestra.run.smithi082.stdout:(18/152): ceph-test-19.0.0-3450.g9a047fe7.el9.x 14 MB/s | 49 MB 00:03 2024-05-03T17:09:43.533 INFO:teuthology.orchestra.run.smithi070.stdout:(77/152): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-05-03T17:09:43.534 INFO:teuthology.orchestra.run.smithi067.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3450.g9a 5.4 MB/s | 274 kB 00:00 2024-05-03T17:09:43.536 INFO:teuthology.orchestra.run.smithi082.stdout:(19/152): python3-ceph-argparse-19.0.0-3450.g9a 913 kB/s | 45 kB 00:00 2024-05-03T17:09:43.558 INFO:teuthology.orchestra.run.smithi070.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 191 kB/s | 19 kB 00:00 2024-05-03T17:09:43.568 INFO:teuthology.orchestra.run.smithi067.stdout:(33/152): ceph-mgr-rook-19.0.0-3450.g9a047fe7.e 1.4 MB/s | 49 kB 00:00 2024-05-03T17:09:43.592 INFO:teuthology.orchestra.run.smithi070.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-05-03T17:09:43.595 INFO:teuthology.orchestra.run.smithi082.stdout:(20/152): python3-ceph-common-19.0.0-3450.g9a04 2.3 MB/s | 135 kB 00:00 2024-05-03T17:09:43.610 INFO:teuthology.orchestra.run.smithi067.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3450.g9 394 kB/s | 16 kB 00:00 2024-05-03T17:09:43.617 INFO:teuthology.orchestra.run.smithi070.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 591 kB/s | 35 kB 00:00 2024-05-03T17:09:43.645 INFO:teuthology.orchestra.run.smithi082.stdout:(21/152): python3-cephfs-19.0.0-3450.g9a047fe7. 3.2 MB/s | 161 kB 00:00 2024-05-03T17:09:43.660 INFO:teuthology.orchestra.run.smithi067.stdout:(35/152): ceph-volume-19.0.0-3450.g9a047fe7.el9 5.3 MB/s | 273 kB 00:00 2024-05-03T17:09:43.693 INFO:teuthology.orchestra.run.smithi070.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-05-03T17:09:43.696 INFO:teuthology.orchestra.run.smithi082.stdout:(22/152): python3-rados-19.0.0-3450.g9a047fe7.e 6.3 MB/s | 320 kB 00:00 2024-05-03T17:09:43.752 INFO:teuthology.orchestra.run.smithi067.stdout:(36/152): cephadm-19.0.0-3450.g9a047fe7.el9.noa 8.1 MB/s | 763 kB 00:00 2024-05-03T17:09:43.754 INFO:teuthology.orchestra.run.smithi082.stdout:(23/152): python3-rbd-19.0.0-3450.g9a047fe7.el9 5.0 MB/s | 298 kB 00:00 2024-05-03T17:09:43.794 INFO:teuthology.orchestra.run.smithi067.stdout:(37/152): cephfs-top-19.0.0-3450.g9a047fe7.el9. 439 kB/s | 18 kB 00:00 2024-05-03T17:09:43.805 INFO:teuthology.orchestra.run.smithi082.stdout:(24/152): python3-rgw-19.0.0-3450.g9a047fe7.el9 2.0 MB/s | 100 kB 00:00 2024-05-03T17:09:43.805 INFO:teuthology.orchestra.run.smithi070.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 222 kB 00:00 2024-05-03T17:09:43.855 INFO:teuthology.orchestra.run.smithi082.stdout:(25/152): rbd-fuse-19.0.0-3450.g9a047fe7.el9.x8 1.7 MB/s | 85 kB 00:00 2024-05-03T17:09:43.877 INFO:teuthology.orchestra.run.smithi070.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 5.1 MB 00:00 2024-05-03T17:09:43.902 INFO:teuthology.orchestra.run.smithi070.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 801 kB/s | 77 kB 00:00 2024-05-03T17:09:43.944 INFO:teuthology.orchestra.run.smithi070.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 631 kB/s | 42 kB 00:00 2024-05-03T17:09:43.969 INFO:teuthology.orchestra.run.smithi070.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 3.9 MB/s | 269 kB 00:00 2024-05-03T17:09:44.011 INFO:teuthology.orchestra.run.smithi070.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-05-03T17:09:44.037 INFO:teuthology.orchestra.run.smithi070.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-05-03T17:09:44.078 INFO:teuthology.orchestra.run.smithi070.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 762 kB/s | 51 kB 00:00 2024-05-03T17:09:44.088 INFO:teuthology.orchestra.run.smithi082.stdout:(26/152): rbd-mirror-19.0.0-3450.g9a047fe7.el9. 13 MB/s | 3.1 MB 00:00 2024-05-03T17:09:44.104 INFO:teuthology.orchestra.run.smithi070.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 802 kB/s | 54 kB 00:00 2024-05-03T17:09:44.139 INFO:teuthology.orchestra.run.smithi082.stdout:(27/152): rbd-nbd-19.0.0-3450.g9a047fe7.el9.x86 3.4 MB/s | 170 kB 00:00 2024-05-03T17:09:44.181 INFO:teuthology.orchestra.run.smithi082.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3450.g 822 kB/s | 34 kB 00:00 2024-05-03T17:09:44.187 INFO:teuthology.orchestra.run.smithi070.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 499 kB/s | 42 kB 00:00 2024-05-03T17:09:44.231 INFO:teuthology.orchestra.run.smithi082.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3450.g9a047fe 2.8 MB/s | 142 kB 00:00 2024-05-03T17:09:44.263 INFO:teuthology.orchestra.run.smithi070.stdout:(92/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 588 kB/s | 44 kB 00:00 2024-05-03T17:09:44.329 INFO:teuthology.orchestra.run.smithi070.stdout:(93/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-05-03T17:09:44.388 INFO:teuthology.orchestra.run.smithi070.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 614 kB/s | 36 kB 00:00 2024-05-03T17:09:44.439 INFO:teuthology.orchestra.run.smithi070.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-05-03T17:09:44.490 INFO:teuthology.orchestra.run.smithi082.stdout:(30/152): ceph-mgr-dashboard-19.0.0-3450.g9a047 14 MB/s | 3.5 MB 00:00 2024-05-03T17:09:44.497 INFO:teuthology.orchestra.run.smithi070.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-05-03T17:09:44.547 INFO:teuthology.orchestra.run.smithi070.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-05-03T17:09:44.606 INFO:teuthology.orchestra.run.smithi070.stdout:(98/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-05-03T17:09:44.653 INFO:teuthology.orchestra.run.smithi067.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 94 kB/s | 80 kB 00:00 2024-05-03T17:09:44.698 INFO:teuthology.orchestra.run.smithi082.stdout:(31/152): ceph-common-19.0.0-3450.g9a047fe7.el9 2.2 MB/s | 21 MB 00:09 2024-05-03T17:09:44.773 INFO:teuthology.orchestra.run.smithi070.stdout:(99/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-05-03T17:09:44.774 INFO:teuthology.orchestra.run.smithi082.stdout:(32/152): librgw2-19.0.0-3450.g9a047fe7.el9.x86 2.7 MB/s | 5.4 MB 00:01 2024-05-03T17:09:44.798 INFO:teuthology.orchestra.run.smithi070.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 332 kB/s | 64 kB 00:00 2024-05-03T17:09:44.823 INFO:teuthology.orchestra.run.smithi070.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 898 kB/s | 45 kB 00:00 2024-05-03T17:09:44.824 INFO:teuthology.orchestra.run.smithi082.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3450.g9a 2.1 MB/s | 274 kB 00:00 2024-05-03T17:09:44.849 INFO:teuthology.orchestra.run.smithi082.stdout:(34/152): ceph-mgr-rook-19.0.0-3450.g9a047fe7.e 657 kB/s | 49 kB 00:00 2024-05-03T17:09:44.882 INFO:teuthology.orchestra.run.smithi082.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3450.g9 281 kB/s | 16 kB 00:00 2024-05-03T17:09:44.919 INFO:teuthology.orchestra.run.smithi067.stdout:(39/152): ceph-radosgw-19.0.0-3450.g9a047fe7.el 980 kB/s | 11 MB 00:11 2024-05-03T17:09:44.957 INFO:teuthology.orchestra.run.smithi070.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 140 kB/s | 22 kB 00:00 2024-05-03T17:09:45.024 INFO:teuthology.orchestra.run.smithi082.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 14 MB/s | 7.4 MB 00:00 2024-05-03T17:09:45.051 INFO:teuthology.orchestra.run.smithi082.stdout:(37/152): ceph-volume-19.0.0-3450.g9a047fe7.el9 1.3 MB/s | 273 kB 00:00 2024-05-03T17:09:45.075 INFO:teuthology.orchestra.run.smithi082.stdout:(38/152): cephfs-top-19.0.0-3450.g9a047fe7.el9. 364 kB/s | 18 kB 00:00 2024-05-03T17:09:45.117 INFO:teuthology.orchestra.run.smithi082.stdout:(39/152): cephadm-19.0.0-3450.g9a047fe7.el9.noa 3.2 MB/s | 763 kB 00:00 2024-05-03T17:09:45.191 INFO:teuthology.orchestra.run.smithi070.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.3 MB/s | 863 kB 00:00 2024-05-03T17:09:45.209 INFO:teuthology.orchestra.run.smithi082.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 506 kB/s | 80 kB 00:00 2024-05-03T17:09:45.259 INFO:teuthology.orchestra.run.smithi082.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-05-03T17:09:45.293 INFO:teuthology.orchestra.run.smithi082.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-05-03T17:09:45.327 INFO:teuthology.orchestra.run.smithi082.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.1 MB/s | 253 kB 00:00 2024-05-03T17:09:45.345 INFO:teuthology.orchestra.run.smithi067.stdout:(40/152): librgw2-19.0.0-3450.g9a047fe7.el9.x86 1.3 MB/s | 5.4 MB 00:04 2024-05-03T17:09:45.352 INFO:teuthology.orchestra.run.smithi082.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-05-03T17:09:45.377 INFO:teuthology.orchestra.run.smithi082.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-05-03T17:09:45.402 INFO:teuthology.orchestra.run.smithi082.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-05-03T17:09:45.428 INFO:teuthology.orchestra.run.smithi082.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-05-03T17:09:45.478 INFO:teuthology.orchestra.run.smithi082.stdout:(48/152): python3-cryptography-36.0.1-4.el9.x86 5.6 MB/s | 1.2 MB 00:00 2024-05-03T17:09:45.653 INFO:teuthology.orchestra.run.smithi082.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 611 kB/s | 107 kB 00:00 2024-05-03T17:09:45.687 INFO:teuthology.orchestra.run.smithi082.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 689 kB 00:00 2024-05-03T17:09:45.712 INFO:teuthology.orchestra.run.smithi082.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 368 kB/s | 21 kB 00:00 2024-05-03T17:09:45.745 INFO:teuthology.orchestra.run.smithi067.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 735 kB/s | 803 kB 00:01 2024-05-03T17:09:45.771 INFO:teuthology.orchestra.run.smithi082.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 561 kB/s | 33 kB 00:00 2024-05-03T17:09:45.796 INFO:teuthology.orchestra.run.smithi082.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 2.8 MB/s | 313 kB 00:00 2024-05-03T17:09:45.820 INFO:teuthology.orchestra.run.smithi067.stdout:(42/152): libquadmath-11.4.1-3.el9.x86_64.rpm 224 kB/s | 195 kB 00:00 2024-05-03T17:09:45.821 INFO:teuthology.orchestra.run.smithi082.stdout:(54/152): bison-3.7.4-5.el9.x86_64.rpm 2.3 MB/s | 943 kB 00:00 2024-05-03T17:09:45.847 INFO:teuthology.orchestra.run.smithi082.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 362 kB/s | 18 kB 00:00 2024-05-03T17:09:45.872 INFO:teuthology.orchestra.run.smithi082.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-05-03T17:09:45.897 INFO:teuthology.orchestra.run.smithi082.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-05-03T17:09:45.922 INFO:teuthology.orchestra.run.smithi082.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 899 kB/s | 45 kB 00:00 2024-05-03T17:09:46.015 INFO:teuthology.orchestra.run.smithi082.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-05-03T17:09:46.024 INFO:teuthology.orchestra.run.smithi070.stdout:(104/152): ceph-osd-19.0.0-3450.g9a047fe7.el9.x 1.2 MB/s | 17 MB 00:14 2024-05-03T17:09:46.040 INFO:teuthology.orchestra.run.smithi082.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-05-03T17:09:46.066 INFO:teuthology.orchestra.run.smithi082.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 507 kB/s | 25 kB 00:00 2024-05-03T17:09:46.090 INFO:teuthology.orchestra.run.smithi082.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 946 kB/s | 47 kB 00:00 2024-05-03T17:09:46.141 INFO:teuthology.orchestra.run.smithi070.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 167 kB/s | 19 kB 00:00 2024-05-03T17:09:46.149 INFO:teuthology.orchestra.run.smithi082.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 3.2 MB/s | 188 kB 00:00 2024-05-03T17:09:46.174 INFO:teuthology.orchestra.run.smithi082.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-05-03T17:09:46.179 INFO:teuthology.orchestra.run.smithi067.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 304 kB/s | 253 kB 00:00 2024-05-03T17:09:46.183 INFO:teuthology.orchestra.run.smithi070.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 311 kB/s | 309 kB 00:00 2024-05-03T17:09:46.199 INFO:teuthology.orchestra.run.smithi082.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 35 kB 00:00 2024-05-03T17:09:46.275 INFO:teuthology.orchestra.run.smithi082.stdout:(66/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.9 MB/s | 3.0 MB 00:00 2024-05-03T17:09:46.283 INFO:teuthology.orchestra.run.smithi070.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 413 kB/s | 548 kB 00:01 2024-05-03T17:09:46.384 INFO:teuthology.orchestra.run.smithi070.stdout:(108/152): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-05-03T17:09:46.426 INFO:teuthology.orchestra.run.smithi070.stdout:(109/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 107 kB/s | 26 kB 00:00 2024-05-03T17:09:46.451 INFO:teuthology.orchestra.run.smithi070.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-05-03T17:09:46.493 INFO:teuthology.orchestra.run.smithi070.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-05-03T17:09:46.550 INFO:teuthology.orchestra.run.smithi082.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 5.8 MB/s | 1.6 MB 00:00 2024-05-03T17:09:46.596 INFO:teuthology.orchestra.run.smithi067.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 137 kB/s | 106 kB 00:00 2024-05-03T17:09:46.652 INFO:teuthology.orchestra.run.smithi070.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-05-03T17:09:46.746 INFO:teuthology.orchestra.run.smithi067.stdout:(45/152): python3-cryptography-36.0.1-4.el9.x86 1.2 MB/s | 1.2 MB 00:01 2024-05-03T17:09:46.760 INFO:teuthology.orchestra.run.smithi070.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 401 kB/s | 43 kB 00:00 2024-05-03T17:09:46.784 INFO:teuthology.orchestra.run.smithi082.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 7.6 MB/s | 4.4 MB 00:00 2024-05-03T17:09:46.819 INFO:teuthology.orchestra.run.smithi070.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 550 kB/s | 32 kB 00:00 2024-05-03T17:09:46.825 INFO:teuthology.orchestra.run.smithi082.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 3.2 MB/s | 911 kB 00:00 2024-05-03T17:09:46.844 INFO:teuthology.orchestra.run.smithi070.stdout:(115/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.1 MB/s | 839 kB 00:00 2024-05-03T17:09:46.859 INFO:teuthology.orchestra.run.smithi082.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-05-03T17:09:46.886 INFO:teuthology.orchestra.run.smithi070.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-05-03T17:09:46.927 INFO:teuthology.orchestra.run.smithi082.stdout:(71/152): openblas-openmp-0.3.21-2.el9.x86_64.r 6.3 MB/s | 4.7 MB 00:00 2024-05-03T17:09:46.945 INFO:teuthology.orchestra.run.smithi070.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-05-03T17:09:46.970 INFO:teuthology.orchestra.run.smithi070.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-05-03T17:09:46.971 INFO:teuthology.orchestra.run.smithi067.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 171 kB/s | 135 kB 00:00 2024-05-03T17:09:46.977 INFO:teuthology.orchestra.run.smithi082.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-05-03T17:09:47.027 INFO:teuthology.orchestra.run.smithi082.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 6.0 MB/s | 1.0 MB 00:00 2024-05-03T17:09:47.086 INFO:teuthology.orchestra.run.smithi082.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-05-03T17:09:47.144 INFO:teuthology.orchestra.run.smithi082.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-05-03T17:09:47.195 INFO:teuthology.orchestra.run.smithi070.stdout:(119/152): libarrow-9.0.0-11.el9.x86_64.rpm 4.2 MB/s | 4.4 MB 00:01 2024-05-03T17:09:47.203 INFO:teuthology.orchestra.run.smithi082.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 439 kB/s | 26 kB 00:00 2024-05-03T17:09:47.237 INFO:teuthology.orchestra.run.smithi070.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 132 kB/s | 35 kB 00:00 2024-05-03T17:09:47.253 INFO:teuthology.orchestra.run.smithi082.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 384 kB/s | 19 kB 00:00 2024-05-03T17:09:47.271 INFO:teuthology.orchestra.run.smithi067.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 2.2 MB/s | 689 kB 00:00 2024-05-03T17:09:47.287 INFO:teuthology.orchestra.run.smithi070.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 2.2 MB/s | 206 kB 00:00 2024-05-03T17:09:47.312 INFO:teuthology.orchestra.run.smithi082.stdout:(78/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-05-03T17:09:47.321 INFO:teuthology.orchestra.run.smithi070.stdout:(122/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 806 kB/s | 303 kB 00:00 2024-05-03T17:09:47.347 INFO:teuthology.orchestra.run.smithi067.stdout:(48/152): python3-requests-2.25.1-8.el9.noarch. 166 kB/s | 125 kB 00:00 2024-05-03T17:09:47.370 INFO:teuthology.orchestra.run.smithi082.stdout:(79/152): python3-markupsafe-1.1.1-12.el9.x86_6 595 kB/s | 35 kB 00:00 2024-05-03T17:09:47.380 INFO:teuthology.orchestra.run.smithi067.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 8.5 MB/s | 943 kB 00:00 2024-05-03T17:09:47.431 INFO:teuthology.orchestra.run.smithi067.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 431 kB/s | 21 kB 00:00 2024-05-03T17:09:47.455 INFO:teuthology.orchestra.run.smithi070.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 9.4 MB/s | 2.0 MB 00:00 2024-05-03T17:09:47.481 INFO:teuthology.orchestra.run.smithi067.stdout:(51/152): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-05-03T17:09:47.498 INFO:teuthology.orchestra.run.smithi070.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 691 kB/s | 144 kB 00:00 2024-05-03T17:09:47.531 INFO:teuthology.orchestra.run.smithi067.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 657 kB/s | 33 kB 00:00 2024-05-03T17:09:47.556 INFO:teuthology.orchestra.run.smithi067.stdout:(53/152): boost-program-options-1.75.0-8.el9.x8 512 kB/s | 107 kB 00:00 2024-05-03T17:09:47.607 INFO:teuthology.orchestra.run.smithi067.stdout:(54/152): python3-urllib3-1.26.5-5.el9.noarch.r 249 kB/s | 215 kB 00:00 2024-05-03T17:09:47.712 INFO:teuthology.orchestra.run.smithi082.stdout:(80/152): python3-babel-2.9.1-2.el9.noarch.rpm 8.1 MB/s | 6.0 MB 00:00 2024-05-03T17:09:47.740 INFO:teuthology.orchestra.run.smithi067.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-05-03T17:09:47.771 INFO:teuthology.orchestra.run.smithi082.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-05-03T17:09:47.824 INFO:teuthology.orchestra.run.smithi067.stdout:(56/152): flexiblas-openblas-openmp-3.0.4-8.el9 68 kB/s | 18 kB 00:00 2024-05-03T17:09:47.829 INFO:teuthology.orchestra.run.smithi082.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-05-03T17:09:47.841 INFO:teuthology.orchestra.run.smithi067.stdout:(57/152): libaio-devel-0.3.111-13.el9.x86_64.rp 48 kB/s | 11 kB 00:00 2024-05-03T17:09:47.866 INFO:teuthology.orchestra.run.smithi067.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-05-03T17:09:47.880 INFO:teuthology.orchestra.run.smithi082.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-03T17:09:47.900 INFO:teuthology.orchestra.run.smithi067.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 599 kB/s | 45 kB 00:00 2024-05-03T17:09:47.950 INFO:teuthology.orchestra.run.smithi082.stdout:(84/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.8 MB/s | 5.1 MB 00:00 2024-05-03T17:09:47.972 INFO:teuthology.orchestra.run.smithi082.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 458 kB/s | 42 kB 00:00 2024-05-03T17:09:48.006 INFO:teuthology.orchestra.run.smithi082.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 4.8 MB/s | 269 kB 00:00 2024-05-03T17:09:48.025 INFO:teuthology.orchestra.run.smithi067.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 3.6 MB/s | 582 kB 00:00 2024-05-03T17:09:48.031 INFO:teuthology.orchestra.run.smithi082.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-05-03T17:09:48.059 INFO:teuthology.orchestra.run.smithi067.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 160 kB/s | 25 kB 00:00 2024-05-03T17:09:48.064 INFO:teuthology.orchestra.run.smithi082.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 4.6 MB/s | 279 kB 00:00 2024-05-03T17:09:48.076 INFO:teuthology.orchestra.run.smithi067.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 939 kB/s | 47 kB 00:00 2024-05-03T17:09:48.090 INFO:teuthology.orchestra.run.smithi082.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 873 kB/s | 51 kB 00:00 2024-05-03T17:09:48.110 INFO:teuthology.orchestra.run.smithi067.stdout:(63/152): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-05-03T17:09:48.115 INFO:teuthology.orchestra.run.smithi082.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-05-03T17:09:48.151 INFO:teuthology.orchestra.run.smithi067.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 850 kB/s | 35 kB 00:00 2024-05-03T17:09:48.165 INFO:teuthology.orchestra.run.smithi082.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 833 kB/s | 42 kB 00:00 2024-05-03T17:09:48.193 INFO:teuthology.orchestra.run.smithi067.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 188 kB 00:00 2024-05-03T17:09:48.249 INFO:teuthology.orchestra.run.smithi082.stdout:(92/152): postgresql-server-13.14-1.el9.x86_64. 4.1 MB/s | 5.8 MB 00:01 2024-05-03T17:09:48.269 INFO:teuthology.orchestra.run.smithi067.stdout:(66/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-05-03T17:09:48.274 INFO:teuthology.orchestra.run.smithi082.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 406 kB/s | 44 kB 00:00 2024-05-03T17:09:48.333 INFO:teuthology.orchestra.run.smithi082.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 617 kB/s | 36 kB 00:00 2024-05-03T17:09:48.335 INFO:teuthology.orchestra.run.smithi067.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-05-03T17:09:48.358 INFO:teuthology.orchestra.run.smithi082.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.0 MB/s | 221 kB 00:00 2024-05-03T17:09:48.392 INFO:teuthology.orchestra.run.smithi082.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 809 kB/s | 47 kB 00:00 2024-05-03T17:09:48.436 INFO:teuthology.orchestra.run.smithi067.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 911 kB 00:00 2024-05-03T17:09:48.450 INFO:teuthology.orchestra.run.smithi082.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 989 kB/s | 58 kB 00:00 2024-05-03T17:09:48.476 INFO:teuthology.orchestra.run.smithi082.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.5 MB/s | 305 kB 00:00 2024-05-03T17:09:48.503 INFO:teuthology.orchestra.run.smithi067.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.0 MB/s | 133 kB 00:00 2024-05-03T17:09:48.510 INFO:teuthology.orchestra.run.smithi082.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.6 MB/s | 94 kB 00:00 2024-05-03T17:09:48.535 INFO:teuthology.orchestra.run.smithi082.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 64 kB 00:00 2024-05-03T17:09:48.568 INFO:teuthology.orchestra.run.smithi082.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 767 kB/s | 45 kB 00:00 2024-05-03T17:09:48.669 INFO:teuthology.orchestra.run.smithi082.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 167 kB/s | 22 kB 00:00 2024-05-03T17:09:48.780 INFO:teuthology.orchestra.run.smithi070.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 13 kB/s | 18 kB 00:01 2024-05-03T17:09:48.786 INFO:teuthology.orchestra.run.smithi082.stdout:(103/152): protobuf-compiler-3.14.0-13.el9.x86_ 3.9 MB/s | 863 kB 00:00 2024-05-03T17:09:48.797 INFO:teuthology.orchestra.run.smithi070.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 18 kB/s | 23 kB 00:01 2024-05-03T17:09:48.814 INFO:teuthology.orchestra.run.smithi070.stdout:(127/152): python3-jaraco-8.2.1-3.el9.noarch.rp 7.2 kB/s | 11 kB 00:01 2024-05-03T17:09:48.820 INFO:teuthology.orchestra.run.smithi067.stdout:(70/152): postgresql-server-13.14-1.el9.x86_64. 18 MB/s | 5.8 MB 00:00 2024-05-03T17:09:48.847 INFO:teuthology.orchestra.run.smithi070.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 394 kB/s | 20 kB 00:00 2024-05-03T17:09:48.973 INFO:teuthology.orchestra.run.smithi070.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 370 kB/s | 46 kB 00:00 2024-05-03T17:09:48.990 INFO:teuthology.orchestra.run.smithi070.stdout:(130/152): python3-jaraco-functools-3.5.0-2.el9 101 kB/s | 19 kB 00:00 2024-05-03T17:09:49.056 INFO:teuthology.orchestra.run.smithi070.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 946 kB/s | 79 kB 00:00 2024-05-03T17:09:49.078 INFO:teuthology.orchestra.run.smithi067.stdout:(71/152): postgresql-13.14-1.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-05-03T17:09:49.082 INFO:teuthology.orchestra.run.smithi070.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 627 kB/s | 58 kB 00:00 2024-05-03T17:09:49.116 INFO:teuthology.orchestra.run.smithi070.stdout:(133/152): python3-kubernetes-26.1.0-2.el9.noar 3.4 MB/s | 1.0 MB 00:00 2024-05-03T17:09:49.141 INFO:teuthology.orchestra.run.smithi070.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-05-03T17:09:49.145 INFO:teuthology.orchestra.run.smithi067.stdout:(72/152): protobuf-3.14.0-13.el9.x86_64.rpm 3.1 MB/s | 1.0 MB 00:00 2024-05-03T17:09:49.153 INFO:teuthology.orchestra.run.smithi082.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 842 kB/s | 309 kB 00:00 2024-05-03T17:09:49.178 INFO:teuthology.orchestra.run.smithi082.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-05-03T17:09:49.191 INFO:teuthology.orchestra.run.smithi070.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-05-03T17:09:49.216 INFO:teuthology.orchestra.run.smithi070.stdout:(136/152): python3-portend-3.1.0-2.el9.noarch.r 122 kB/s | 16 kB 00:00 2024-05-03T17:09:49.245 INFO:teuthology.orchestra.run.smithi082.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-05-03T17:09:49.287 INFO:teuthology.orchestra.run.smithi067.stdout:(73/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 4.0 MB/s | 4.4 MB 00:01 2024-05-03T17:09:49.338 INFO:teuthology.orchestra.run.smithi067.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-05-03T17:09:49.438 INFO:teuthology.orchestra.run.smithi067.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-05-03T17:09:49.445 INFO:teuthology.orchestra.run.smithi082.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 130 kB/s | 26 kB 00:00 2024-05-03T17:09:49.512 INFO:teuthology.orchestra.run.smithi082.stdout:(108/152): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-05-03T17:09:49.521 INFO:teuthology.orchestra.run.smithi067.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-05-03T17:09:49.538 INFO:teuthology.orchestra.run.smithi082.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 529 kB/s | 49 kB 00:00 2024-05-03T17:09:49.546 INFO:teuthology.orchestra.run.smithi067.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 236 kB/s | 26 kB 00:00 2024-05-03T17:09:49.572 INFO:teuthology.orchestra.run.smithi067.stdout:(78/152): python3-attrs-20.3.0-7.el9.noarch.rpm 177 kB/s | 87 kB 00:00 2024-05-03T17:09:49.597 INFO:teuthology.orchestra.run.smithi067.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 254 kB/s | 19 kB 00:00 2024-05-03T17:09:49.664 INFO:teuthology.orchestra.run.smithi067.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 378 kB/s | 35 kB 00:00 2024-05-03T17:09:49.688 INFO:teuthology.orchestra.run.smithi082.stdout:(110/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-05-03T17:09:49.704 INFO:teuthology.orchestra.run.smithi082.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 351 kB/s | 67 kB 00:00 2024-05-03T17:09:49.714 INFO:teuthology.orchestra.run.smithi067.stdout:(81/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-05-03T17:09:49.730 INFO:teuthology.orchestra.run.smithi082.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 785 kB/s | 151 kB 00:00 2024-05-03T17:09:49.798 INFO:teuthology.orchestra.run.smithi067.stdout:(82/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-05-03T17:09:49.823 INFO:teuthology.orchestra.run.smithi067.stdout:(83/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 250 kB 00:00 2024-05-03T17:09:49.848 INFO:teuthology.orchestra.run.smithi067.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-05-03T17:09:49.850 INFO:teuthology.orchestra.run.smithi070.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 47 kB/s | 31 kB 00:00 2024-05-03T17:09:49.880 INFO:teuthology.orchestra.run.smithi082.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 3.1 MB/s | 548 kB 00:00 2024-05-03T17:09:49.883 INFO:teuthology.orchestra.run.smithi070.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 283 kB/s | 188 kB 00:00 2024-05-03T17:09:49.905 INFO:teuthology.orchestra.run.smithi082.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 248 kB/s | 43 kB 00:00 2024-05-03T17:09:49.915 INFO:teuthology.orchestra.run.smithi067.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 660 kB/s | 77 kB 00:00 2024-05-03T17:09:49.941 INFO:teuthology.orchestra.run.smithi067.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 360 kB/s | 42 kB 00:00 2024-05-03T17:09:49.955 INFO:teuthology.orchestra.run.smithi082.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 427 kB/s | 32 kB 00:00 2024-05-03T17:09:49.966 INFO:teuthology.orchestra.run.smithi067.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 2.2 MB/s | 269 kB 00:00 2024-05-03T17:09:49.989 INFO:teuthology.orchestra.run.smithi082.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 179 kB/s | 15 kB 00:00 2024-05-03T17:09:50.024 INFO:teuthology.orchestra.run.smithi067.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-05-03T17:09:50.034 INFO:teuthology.orchestra.run.smithi070.stdout:(139/152): python3-pyperclip-1.8.2-2.el9.noarch 32 kB/s | 28 kB 00:00 2024-05-03T17:09:50.039 INFO:teuthology.orchestra.run.smithi082.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-05-03T17:09:50.050 INFO:teuthology.orchestra.run.smithi067.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 610 kB/s | 51 kB 00:00 2024-05-03T17:09:50.060 INFO:teuthology.orchestra.run.smithi070.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 204 kB/s | 36 kB 00:00 2024-05-03T17:09:50.075 INFO:teuthology.orchestra.run.smithi067.stdout:(90/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.0 MB/s | 279 kB 00:00 2024-05-03T17:09:50.100 INFO:teuthology.orchestra.run.smithi067.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 709 kB/s | 54 kB 00:00 2024-05-03T17:09:50.142 INFO:teuthology.orchestra.run.smithi067.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 620 kB/s | 42 kB 00:00 2024-05-03T17:09:50.145 INFO:teuthology.orchestra.run.smithi082.stdout:(118/152): python3-cherrypy-18.6.1-2.el9.noarch 2.2 MB/s | 358 kB 00:00 2024-05-03T17:09:50.168 INFO:teuthology.orchestra.run.smithi067.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 656 kB/s | 44 kB 00:00 2024-05-03T17:09:50.174 INFO:teuthology.orchestra.run.smithi082.stdout:(119/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.7 MB/s | 839 kB 00:00 2024-05-03T17:09:50.199 INFO:teuthology.orchestra.run.smithi082.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.9 MB/s | 303 kB 00:00 2024-05-03T17:09:50.224 INFO:teuthology.orchestra.run.smithi082.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 448 kB/s | 35 kB 00:00 2024-05-03T17:09:50.318 INFO:teuthology.orchestra.run.smithi067.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 1.2 MB/s | 221 kB 00:00 2024-05-03T17:09:50.344 INFO:teuthology.orchestra.run.smithi067.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 205 kB/s | 36 kB 00:00 2024-05-03T17:09:50.349 INFO:teuthology.orchestra.run.smithi082.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 1.1 MB/s | 206 kB 00:00 2024-05-03T17:09:50.368 INFO:teuthology.orchestra.run.smithi067.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 949 kB/s | 47 kB 00:00 2024-05-03T17:09:50.416 INFO:teuthology.orchestra.run.smithi082.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 9.4 MB/s | 2.0 MB 00:00 2024-05-03T17:09:50.436 INFO:teuthology.orchestra.run.smithi067.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 3.2 MB/s | 305 kB 00:00 2024-05-03T17:09:50.441 INFO:teuthology.orchestra.run.smithi082.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-05-03T17:09:50.461 INFO:teuthology.orchestra.run.smithi067.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 628 kB/s | 58 kB 00:00 2024-05-03T17:09:50.467 INFO:teuthology.orchestra.run.smithi082.stdout:(125/152): python3-grpcio-tools-1.46.7-10.el9.x 596 kB/s | 144 kB 00:00 2024-05-03T17:09:50.500 INFO:teuthology.orchestra.run.smithi082.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 211 kB/s | 18 kB 00:00 2024-05-03T17:09:50.526 INFO:teuthology.orchestra.run.smithi082.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 275 kB/s | 23 kB 00:00 2024-05-03T17:09:50.551 INFO:teuthology.orchestra.run.smithi082.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 231 kB/s | 19 kB 00:00 2024-05-03T17:09:50.576 INFO:teuthology.orchestra.run.smithi082.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 259 kB/s | 20 kB 00:00 2024-05-03T17:09:50.635 INFO:teuthology.orchestra.run.smithi082.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-05-03T17:09:50.645 INFO:teuthology.orchestra.run.smithi067.stdout:(99/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 27 MB/s | 16 MB 00:00 2024-05-03T17:09:50.669 INFO:teuthology.orchestra.run.smithi082.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 7.2 MB/s | 1.0 MB 00:00 2024-05-03T17:09:50.670 INFO:teuthology.orchestra.run.smithi067.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 305 kB/s | 64 kB 00:00 2024-05-03T17:09:50.694 INFO:teuthology.orchestra.run.smithi082.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 673 kB/s | 79 kB 00:00 2024-05-03T17:09:50.695 INFO:teuthology.orchestra.run.smithi067.stdout:(101/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 362 kB/s | 94 kB 00:00 2024-05-03T17:09:50.719 INFO:teuthology.orchestra.run.smithi082.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 687 kB/s | 58 kB 00:00 2024-05-03T17:09:50.761 INFO:teuthology.orchestra.run.smithi082.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.9 MB/s | 272 kB 00:00 2024-05-03T17:09:50.786 INFO:teuthology.orchestra.run.smithi082.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 178 kB/s | 16 kB 00:00 2024-05-03T17:09:50.828 INFO:teuthology.orchestra.run.smithi082.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 825 kB/s | 90 kB 00:00 2024-05-03T17:09:50.854 INFO:teuthology.orchestra.run.smithi082.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 306 kB/s | 28 kB 00:00 2024-05-03T17:09:50.870 INFO:teuthology.orchestra.run.smithi067.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 111 kB/s | 22 kB 00:00 2024-05-03T17:09:50.887 INFO:teuthology.orchestra.run.smithi082.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-05-03T17:09:50.921 INFO:teuthology.orchestra.run.smithi082.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-05-03T17:09:50.946 INFO:teuthology.orchestra.run.smithi082.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 642 kB/s | 59 kB 00:00 2024-05-03T17:09:50.954 INFO:teuthology.orchestra.run.smithi067.stdout:(103/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 145 kB/s | 45 kB 00:00 2024-05-03T17:09:50.971 INFO:teuthology.orchestra.run.smithi082.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 427 kB/s | 36 kB 00:00 2024-05-03T17:09:51.004 INFO:teuthology.orchestra.run.smithi067.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-05-03T17:09:51.005 INFO:teuthology.orchestra.run.smithi082.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 607 kB/s | 51 kB 00:00 2024-05-03T17:09:51.038 INFO:teuthology.orchestra.run.smithi082.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-05-03T17:09:51.064 INFO:teuthology.orchestra.run.smithi082.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 974 kB/s | 90 kB 00:00 2024-05-03T17:09:51.106 INFO:teuthology.orchestra.run.smithi082.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-05-03T17:09:51.131 INFO:teuthology.orchestra.run.smithi082.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-05-03T17:09:51.137 INFO:teuthology.orchestra.run.smithi067.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 2.0 MB/s | 548 kB 00:00 2024-05-03T17:09:51.163 INFO:teuthology.orchestra.run.smithi067.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.5 MB/s | 309 kB 00:00 2024-05-03T17:09:51.173 INFO:teuthology.orchestra.run.smithi082.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-05-03T17:09:51.206 INFO:teuthology.orchestra.run.smithi067.stdout:(107/152): grpc-data-1.46.7-10.el9.noarch.rpm 96 kB/s | 19 kB 00:00 2024-05-03T17:09:51.323 INFO:teuthology.orchestra.run.smithi082.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-05-03T17:09:51.347 INFO:teuthology.orchestra.run.smithi067.stdout:(108/152): libarrow-9.0.0-11.el9.x86_64.rpm 21 MB/s | 4.4 MB 00:00 2024-05-03T17:09:51.372 INFO:teuthology.orchestra.run.smithi067.stdout:(109/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 158 kB/s | 26 kB 00:00 2024-05-03T17:09:51.397 INFO:teuthology.orchestra.run.smithi067.stdout:(110/152): liboath-2.6.7-2.el9.x86_64.rpm 257 kB/s | 49 kB 00:00 2024-05-03T17:09:51.422 INFO:teuthology.orchestra.run.smithi067.stdout:(111/152): libunwind-1.6.2-1.el9.x86_64.rpm 897 kB/s | 67 kB 00:00 2024-05-03T17:09:51.448 INFO:teuthology.orchestra.run.smithi067.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-05-03T17:09:51.473 INFO:teuthology.orchestra.run.smithi082.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 9.0 MB/s | 1.3 MB 00:00 2024-05-03T17:09:51.481 INFO:teuthology.orchestra.run.smithi067.stdout:(113/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 9.8 MB/s | 839 kB 00:00 2024-05-03T17:09:51.506 INFO:teuthology.orchestra.run.smithi067.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 6.5 MB/s | 548 kB 00:00 2024-05-03T17:09:51.532 INFO:teuthology.orchestra.run.smithi067.stdout:(115/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 534 kB/s | 43 kB 00:00 2024-05-03T17:09:51.549 INFO:teuthology.orchestra.run.smithi082.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 32 MB/s | 2.4 MB 00:00 2024-05-03T17:09:51.557 INFO:teuthology.orchestra.run.smithi067.stdout:(116/152): python3-cachetools-4.2.4-1.el9.noarc 429 kB/s | 32 kB 00:00 2024-05-03T17:09:51.582 INFO:teuthology.orchestra.run.smithi067.stdout:(117/152): python3-certifi-2021.10.8-2.el9.noar 300 kB/s | 15 kB 00:00 2024-05-03T17:09:51.607 INFO:teuthology.orchestra.run.smithi067.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-05-03T17:09:51.641 INFO:teuthology.orchestra.run.smithi067.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-05-03T17:09:51.666 INFO:teuthology.orchestra.run.smithi067.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.1 MB/s | 303 kB 00:00 2024-05-03T17:09:51.691 INFO:teuthology.orchestra.run.smithi067.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 426 kB/s | 35 kB 00:00 2024-05-03T17:09:51.716 INFO:teuthology.orchestra.run.smithi067.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-05-03T17:09:51.740 INFO:teuthology.orchestra.run.smithi082.stdout:(151/152): librados2-19.0.0-3450.g9a047fe7.el9. 5.6 MB/s | 3.4 MB 00:00 2024-05-03T17:09:51.775 INFO:teuthology.orchestra.run.smithi067.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 25 MB/s | 2.0 MB 00:00 2024-05-03T17:09:51.801 INFO:teuthology.orchestra.run.smithi067.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-05-03T17:09:51.818 INFO:teuthology.orchestra.run.smithi067.stdout:(125/152): python3-jaraco-8.2.1-3.el9.noarch.rp 106 kB/s | 11 kB 00:00 2024-05-03T17:09:51.824 INFO:teuthology.orchestra.run.smithi082.stdout:(152/152): librbd1-19.0.0-3450.g9a047fe7.el9.x8 4.8 MB/s | 3.1 MB 00:00 2024-05-03T17:09:51.827 INFO:teuthology.orchestra.run.smithi082.stdout:-------------------------------------------------------------------------------- 2024-05-03T17:09:51.827 INFO:teuthology.orchestra.run.smithi082.stdout:Total 13 MB/s | 222 MB 00:17 2024-05-03T17:09:51.835 INFO:teuthology.orchestra.run.smithi067.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 301 kB/s | 18 kB 00:00 2024-05-03T17:09:51.860 INFO:teuthology.orchestra.run.smithi067.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 557 kB/s | 23 kB 00:00 2024-05-03T17:09:51.877 INFO:teuthology.orchestra.run.smithi067.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 333 kB/s | 19 kB 00:00 2024-05-03T17:09:51.894 INFO:teuthology.orchestra.run.smithi067.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 335 kB/s | 20 kB 00:00 2024-05-03T17:09:51.927 INFO:teuthology.orchestra.run.smithi067.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 15 MB/s | 1.0 MB 00:00 2024-05-03T17:09:51.944 INFO:teuthology.orchestra.run.smithi067.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 969 kB/s | 46 kB 00:00 2024-05-03T17:09:51.969 INFO:teuthology.orchestra.run.smithi067.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 1.1 MB/s | 79 kB 00:00 2024-05-03T17:09:51.986 INFO:teuthology.orchestra.run.smithi067.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 987 kB/s | 58 kB 00:00 2024-05-03T17:09:52.020 INFO:teuthology.orchestra.run.smithi067.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-05-03T17:09:52.037 INFO:teuthology.orchestra.run.smithi067.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-05-03T17:09:52.062 INFO:teuthology.orchestra.run.smithi067.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-05-03T17:09:52.087 INFO:teuthology.orchestra.run.smithi067.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 423 kB/s | 28 kB 00:00 2024-05-03T17:09:52.113 INFO:teuthology.orchestra.run.smithi067.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 616 kB/s | 31 kB 00:00 2024-05-03T17:09:52.138 INFO:teuthology.orchestra.run.smithi067.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-05-03T17:09:52.163 INFO:teuthology.orchestra.run.smithi067.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 789 kB/s | 59 kB 00:00 2024-05-03T17:09:52.230 INFO:teuthology.orchestra.run.smithi067.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 391 kB/s | 36 kB 00:00 2024-05-03T17:09:52.255 INFO:teuthology.orchestra.run.smithi067.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 438 kB/s | 51 kB 00:00 2024-05-03T17:09:52.280 INFO:teuthology.orchestra.run.smithi067.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-05-03T17:09:52.306 INFO:teuthology.orchestra.run.smithi067.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-05-03T17:09:52.339 INFO:teuthology.orchestra.run.smithi067.stdout:(145/152): python3-werkzeug-2.0.3-3.el9.1.noarc 5.0 MB/s | 427 kB 00:00 2024-05-03T17:09:52.364 INFO:teuthology.orchestra.run.smithi067.stdout:(146/152): python3-zc-lockfile-2.0-10.el9.noarc 241 kB/s | 20 kB 00:00 2024-05-03T17:09:52.389 INFO:teuthology.orchestra.run.smithi067.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 2.3 MB/s | 192 kB 00:00 2024-05-03T17:09:52.440 INFO:teuthology.orchestra.run.smithi067.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-05-03T17:09:52.651 INFO:teuthology.orchestra.run.smithi070.stdout:(141/152): python3-rsa-4.9-2.el9.noarch.rpm 21 kB/s | 59 kB 00:02 2024-05-03T17:09:52.676 INFO:teuthology.orchestra.run.smithi070.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 19 kB/s | 51 kB 00:02 2024-05-03T17:09:52.701 INFO:teuthology.orchestra.run.smithi070.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 87 kB/s | 230 kB 00:02 2024-05-03T17:09:52.818 INFO:teuthology.orchestra.run.smithi070.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-05-03T17:09:52.852 INFO:teuthology.orchestra.run.smithi070.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 133 kB/s | 20 kB 00:00 2024-05-03T17:09:52.877 INFO:teuthology.orchestra.run.smithi070.stdout:(146/152): python3-websocket-client-1.2.3-2.el9 397 kB/s | 90 kB 00:00 2024-05-03T17:09:52.946 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction check 2024-05-03T17:09:53.088 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction check succeeded. 2024-05-03T17:09:53.089 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction test 2024-05-03T17:09:53.432 INFO:teuthology.orchestra.run.smithi067.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 1.4 MB/s | 1.3 MB 00:00 2024-05-03T17:09:53.515 INFO:teuthology.orchestra.run.smithi067.stdout:(150/152): librbd1-19.0.0-3450.g9a047fe7.el9.x8 2.8 MB/s | 3.1 MB 00:01 2024-05-03T17:09:53.749 INFO:teuthology.orchestra.run.smithi067.stdout:(151/152): librados2-19.0.0-3450.g9a047fe7.el9. 2.5 MB/s | 3.4 MB 00:01 2024-05-03T17:09:53.794 INFO:teuthology.orchestra.run.smithi070.stdout:(147/152): librados2-19.0.0-3450.g9a047fe7.el9. 3.7 MB/s | 3.4 MB 00:00 2024-05-03T17:09:54.461 INFO:teuthology.orchestra.run.smithi070.stdout:(148/152): librbd1-19.0.0-3450.g9a047fe7.el9.x8 4.7 MB/s | 3.1 MB 00:00 2024-05-03T17:09:54.541 INFO:teuthology.orchestra.run.smithi067.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.2 MB/s | 2.4 MB 00:01 2024-05-03T17:09:54.547 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-05-03T17:09:54.547 INFO:teuthology.orchestra.run.smithi067.stdout:Total 8.9 MB/s | 222 MB 00:25 2024-05-03T17:09:54.811 INFO:teuthology.orchestra.run.smithi070.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 3.9 MB/s | 1.3 MB 00:00 2024-05-03T17:09:55.002 INFO:teuthology.orchestra.run.smithi082.stdout:Transaction test succeeded. 2024-05-03T17:09:55.003 INFO:teuthology.orchestra.run.smithi082.stdout:Running transaction 2024-05-03T17:09:55.578 INFO:teuthology.orchestra.run.smithi070.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 3.1 MB/s | 2.4 MB 00:00 2024-05-03T17:09:55.695 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2024-05-03T17:09:55.853 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2024-05-03T17:09:55.853 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2024-05-03T17:09:57.724 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2024-05-03T17:09:57.724 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2024-05-03T17:09:58.361 INFO:teuthology.orchestra.run.smithi070.stdout:(151/152): re2-20211101-3.el9.x86_64.rpm 35 kB/s | 192 kB 00:05 2024-05-03T17:10:00.895 INFO:teuthology.orchestra.run.smithi082.stdout: Preparing : 1/1 2024-05-03T17:10:00.978 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-05-03T17:10:01.016 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-05-03T17:10:01.155 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-05-03T17:10:01.288 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-05-03T17:10:01.313 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-05-03T17:10:01.357 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-05-03T17:10:01.686 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-05-03T17:10:01.789 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T17:10:01.910 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T17:10:01.940 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T17:10:02.032 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T17:10:02.061 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 10/156 2024-05-03T17:10:02.215 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-05-03T17:10:02.267 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-05-03T17:10:02.294 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T17:10:02.399 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T17:10:02.428 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 14/156 2024-05-03T17:10:02.474 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 14/156 2024-05-03T17:10:02.496 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-05-03T17:10:02.520 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-05-03T17:10:02.663 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-05-03T17:10:02.695 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-05-03T17:10:02.807 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-05-03T17:10:02.900 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-05-03T17:10:02.937 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-05-03T17:10:02.972 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-05-03T17:10:03.046 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-05-03T17:10:03.079 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-05-03T17:10:03.123 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-05-03T17:10:03.155 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-05-03T17:10:03.183 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-05-03T17:10:03.252 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-05-03T17:10:03.305 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-05-03T17:10:03.336 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-05-03T17:10:03.371 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-05-03T17:10:03.446 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-05-03T17:10:03.482 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9. 31/156 2024-05-03T17:10:03.525 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el 32/156 2024-05-03T17:10:03.554 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_6 33/156 2024-05-03T17:10:03.623 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-05-03T17:10:03.668 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-05-03T17:10:03.742 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-05-03T17:10:03.876 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-05-03T17:10:03.929 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-05-03T17:10:04.104 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2024-05-03T17:10:04.207 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-05-03T17:10:04.260 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-05-03T17:10:04.314 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-05-03T17:10:04.355 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-05-03T17:10:04.381 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-05-03T17:10:04.425 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-05-03T17:10:04.454 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-05-03T17:10:04.532 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-05-03T17:10:04.562 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-05-03T17:10:04.596 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-05-03T17:10:04.634 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_6 43/156 2024-05-03T17:10:04.676 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-05-03T17:10:04.783 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-05-03T17:10:04.824 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-05-03T17:10:04.890 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T17:10:04.895 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-05-03T17:10:04.950 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-05-03T17:10:05.015 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T17:10:05.025 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-05-03T17:10:05.049 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T17:10:05.062 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-05-03T17:10:05.099 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-05-03T17:10:05.141 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T17:10:05.153 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-05-03T17:10:05.170 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 10/156 2024-05-03T17:10:05.190 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-05-03T17:10:05.233 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-05-03T17:10:05.274 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-05-03T17:10:05.327 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-05-03T17:10:05.392 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-05-03T17:10:05.409 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-05-03T17:10:05.411 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T17:10:05.447 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-05-03T17:10:05.477 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-05-03T17:10:05.517 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T17:10:05.518 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-05-03T17:10:05.546 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 14/156 2024-05-03T17:10:05.608 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 14/156 2024-05-03T17:10:05.611 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-05-03T17:10:05.630 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-05-03T17:10:05.652 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-05-03T17:10:05.653 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-05-03T17:10:05.692 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-05-03T17:10:05.717 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-05-03T17:10:05.742 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-05-03T17:10:05.816 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-05-03T17:10:05.847 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-05-03T17:10:05.880 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-05-03T17:10:05.942 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-05-03T17:10:05.975 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-05-03T17:10:05.991 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-05-03T17:10:06.003 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-05-03T17:10:06.037 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-05-03T17:10:06.076 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-05-03T17:10:06.104 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-05-03T17:10:06.116 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-05-03T17:10:06.138 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-05-03T17:10:06.146 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-05-03T17:10:06.173 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-05-03T17:10:06.181 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-05-03T17:10:06.214 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-05-03T17:10:06.249 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-05-03T17:10:06.255 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-05-03T17:10:06.289 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-05-03T17:10:06.290 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-05-03T17:10:06.333 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-05-03T17:10:06.365 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-05-03T17:10:06.376 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-05-03T17:10:06.392 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-05-03T17:10:06.464 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-05-03T17:10:06.504 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-05-03T17:10:06.525 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-05-03T17:10:06.543 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-05-03T17:10:06.554 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-05-03T17:10:06.590 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-05-03T17:10:06.598 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-05-03T17:10:06.684 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-05-03T17:10:06.687 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-05-03T17:10:06.725 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9. 31/156 2024-05-03T17:10:06.735 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-05-03T17:10:06.760 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el 32/156 2024-05-03T17:10:06.789 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_6 33/156 2024-05-03T17:10:06.878 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-05-03T17:10:06.920 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-05-03T17:10:06.998 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-05-03T17:10:07.143 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-05-03T17:10:07.199 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-05-03T17:10:07.318 INFO:teuthology.orchestra.run.smithi070.stdout:(152/152): thrift-0.15.0-2.el9.x86_64.rpm 112 kB/s | 1.6 MB 00:14 2024-05-03T17:10:07.320 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-05-03T17:10:07.320 INFO:teuthology.orchestra.run.smithi070.stdout:Total 5.9 MB/s | 222 MB 00:37 2024-05-03T17:10:07.480 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-05-03T17:10:07.650 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-05-03T17:10:07.683 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-05-03T17:10:07.759 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-05-03T17:10:07.782 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-05-03T17:10:07.833 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-05-03T17:10:07.853 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-05-03T17:10:07.888 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-05-03T17:10:07.920 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-05-03T17:10:07.933 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-05-03T17:10:07.960 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-05-03T17:10:07.961 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_6 43/156 2024-05-03T17:10:08.003 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-05-03T17:10:08.137 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-05-03T17:10:08.166 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-05-03T17:10:08.168 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-05-03T17:10:08.230 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-05-03T17:10:08.236 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-05-03T17:10:08.286 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-05-03T17:10:08.361 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-05-03T17:10:08.397 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-05-03T17:10:08.416 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-05-03T17:10:08.434 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-05-03T17:10:08.482 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-05-03T17:10:08.517 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-05-03T17:10:08.559 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-05-03T17:10:08.559 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-05-03T17:10:08.561 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-05-03T17:10:08.602 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-05-03T17:10:08.745 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-05-03T17:10:08.870 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-05-03T17:10:08.916 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-05-03T17:10:08.946 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-05-03T17:10:08.987 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-05-03T17:10:08.988 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-05-03T17:10:09.086 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-05-03T17:10:09.121 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-05-03T17:10:09.161 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-05-03T17:10:09.195 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-05-03T17:10:09.219 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-05-03T17:10:09.297 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-05-03T17:10:09.433 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-05-03T17:10:09.468 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-05-03T17:10:09.496 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-05-03T17:10:09.531 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-05-03T17:10:09.570 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-05-03T17:10:09.602 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-05-03T17:10:09.631 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-05-03T17:10:09.657 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-05-03T17:10:09.691 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-05-03T17:10:09.733 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-05-03T17:10:09.767 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-05-03T17:10:09.856 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-05-03T17:10:09.906 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-05-03T17:10:09.936 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-05-03T17:10:09.984 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-05-03T17:10:10.096 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-05-03T17:10:10.156 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-05-03T17:10:10.196 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-05-03T17:10:10.278 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-05-03T17:10:10.306 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-05-03T17:10:10.348 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-05-03T17:10:10.382 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-05-03T17:10:10.382 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-05-03T17:10:10.660 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-05-03T17:10:10.702 INFO:teuthology.orchestra.run.smithi082.stdout: Upgrading : librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 99/156 2024-05-03T17:10:10.793 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 99/156 2024-05-03T17:10:10.823 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 100/156 2024-05-03T17:10:10.859 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-05-03T17:10:10.918 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-05-03T17:10:10.961 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-05-03T17:10:10.994 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-05-03T17:10:11.097 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-05-03T17:10:11.152 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-05-03T17:10:11.242 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-05-03T17:10:11.279 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-05-03T17:10:11.338 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-05-03T17:10:11.380 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 103/156 2024-05-03T17:10:11.451 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 103/156 2024-05-03T17:10:11.464 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-05-03T17:10:11.473 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 104/156 2024-05-03T17:10:11.493 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-05-03T17:10:11.574 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-05-03T17:10:12.254 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-05-03T17:10:12.371 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-05-03T17:10:13.663 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-05-03T17:10:13.739 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-05-03T17:10:13.766 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-05-03T17:10:13.797 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T17:10:13.799 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-05-03T17:10:13.828 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T17:10:14.128 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-05-03T17:10:14.171 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 99/156 2024-05-03T17:10:14.262 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 99/156 2024-05-03T17:10:14.292 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 100/156 2024-05-03T17:10:14.329 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-05-03T17:10:14.370 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T17:10:14.406 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 106/156 2024-05-03T17:10:14.498 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 106/156 2024-05-03T17:10:14.498 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-03T17:10:14.498 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-03T17:10:14.498 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:14.533 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 107/156 2024-05-03T17:10:14.835 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-05-03T17:10:14.865 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 103/156 2024-05-03T17:10:14.936 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 103/156 2024-05-03T17:10:14.958 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 104/156 2024-05-03T17:10:16.293 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-05-03T17:10:16.388 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-05-03T17:10:16.426 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-05-03T17:10:16.488 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-05-03T17:10:16.522 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-05-03T17:10:16.548 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-05-03T17:10:16.590 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-05-03T17:10:16.909 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-05-03T17:10:17.032 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T17:10:17.136 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T17:10:17.166 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T17:10:17.240 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T17:10:17.279 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 10/156 2024-05-03T17:10:17.311 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T17:10:17.346 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T17:10:17.426 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-05-03T17:10:17.476 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-05-03T17:10:17.504 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T17:10:17.606 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T17:10:17.630 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 14/156 2024-05-03T17:10:17.692 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 14/156 2024-05-03T17:10:17.714 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-05-03T17:10:17.733 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-05-03T17:10:17.919 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T17:10:17.930 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-05-03T17:10:17.964 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-05-03T17:10:17.966 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 106/156 2024-05-03T17:10:18.051 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 106/156 2024-05-03T17:10:18.051 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-03T17:10:18.051 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-03T17:10:18.051 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:18.073 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-05-03T17:10:18.084 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 107/156 2024-05-03T17:10:18.167 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-05-03T17:10:18.205 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-05-03T17:10:18.240 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-05-03T17:10:18.314 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-05-03T17:10:18.356 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-05-03T17:10:18.400 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-05-03T17:10:18.432 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-05-03T17:10:18.460 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-05-03T17:10:18.530 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-05-03T17:10:18.582 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-05-03T17:10:18.612 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-05-03T17:10:18.648 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-05-03T17:10:18.724 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-05-03T17:10:18.758 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9. 31/156 2024-05-03T17:10:18.793 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el 32/156 2024-05-03T17:10:18.814 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_6 33/156 2024-05-03T17:10:18.883 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-05-03T17:10:18.929 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-05-03T17:10:19.003 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-05-03T17:10:19.137 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-05-03T17:10:19.188 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-05-03T17:10:19.713 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-05-03T17:10:19.793 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-05-03T17:10:19.822 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-05-03T17:10:19.865 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-05-03T17:10:19.903 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_6 43/156 2024-05-03T17:10:19.945 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-05-03T17:10:20.096 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-05-03T17:10:20.164 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-05-03T17:10:20.218 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-05-03T17:10:20.291 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-05-03T17:10:20.323 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-05-03T17:10:20.359 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-05-03T17:10:20.405 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-05-03T17:10:20.442 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-05-03T17:10:20.485 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-05-03T17:10:20.535 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-05-03T17:10:20.670 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-05-03T17:10:20.716 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-05-03T17:10:20.746 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-05-03T17:10:20.788 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-05-03T17:10:20.881 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-05-03T17:10:20.912 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-05-03T17:10:20.953 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-05-03T17:10:20.978 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-05-03T17:10:21.003 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-05-03T17:10:21.077 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-05-03T17:10:21.204 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-05-03T17:10:21.235 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-05-03T17:10:21.263 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-05-03T17:10:21.297 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-05-03T17:10:21.336 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-05-03T17:10:21.367 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-05-03T17:10:21.398 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-05-03T17:10:21.424 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-05-03T17:10:21.457 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-05-03T17:10:21.499 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-05-03T17:10:21.533 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-05-03T17:10:21.618 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-05-03T17:10:21.664 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-05-03T17:10:21.695 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-05-03T17:10:21.741 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-05-03T17:10:21.847 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-05-03T17:10:21.896 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-05-03T17:10:22.609 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-05-03T17:10:22.652 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-05-03T17:10:22.685 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-05-03T17:10:22.783 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-05-03T17:10:22.842 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-05-03T17:10:22.921 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-05-03T17:10:22.962 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-05-03T17:10:23.139 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-05-03T17:10:23.168 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-05-03T17:10:23.255 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-05-03T17:10:23.885 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-05-03T17:10:23.999 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-05-03T17:10:25.217 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-05-03T17:10:25.555 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-05-03T17:10:25.583 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-05-03T17:10:25.616 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-05-03T17:10:25.920 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-05-03T17:10:25.962 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 99/156 2024-05-03T17:10:25.973 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 107/156 2024-05-03T17:10:25.974 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /sys 2024-05-03T17:10:25.974 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /proc 2024-05-03T17:10:25.974 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /mnt 2024-05-03T17:10:25.974 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /var/tmp 2024-05-03T17:10:25.974 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /home 2024-05-03T17:10:25.974 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /root 2024-05-03T17:10:25.974 INFO:teuthology.orchestra.run.smithi082.stdout:skipping the directory /tmp 2024-05-03T17:10:25.974 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:26.053 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 99/156 2024-05-03T17:10:26.084 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 100/156 2024-05-03T17:10:26.120 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-05-03T17:10:26.147 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 108/156 2024-05-03T17:10:26.163 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 108/156 2024-05-03T17:10:26.163 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:26.164 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-03T17:10:26.164 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:26.597 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-05-03T17:10:26.631 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 103/156 2024-05-03T17:10:26.703 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 103/156 2024-05-03T17:10:26.733 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 104/156 2024-05-03T17:10:27.650 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 109/156 2024-05-03T17:10:27.676 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 109/156 2024-05-03T17:10:27.676 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:27.676 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-03T17:10:27.677 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-03T17:10:27.677 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-03T17:10:27.677 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:27.901 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 110/156 2024-05-03T17:10:27.923 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 110/156 2024-05-03T17:10:27.923 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:27.923 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-03T17:10:27.924 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-03T17:10:27.924 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-03T17:10:27.924 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:28.360 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 111/156 2024-05-03T17:10:28.383 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 111/156 2024-05-03T17:10:28.383 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:28.383 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-03T17:10:28.383 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-03T17:10:28.383 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-03T17:10:28.383 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:28.481 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-05-03T17:10:28.530 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-05-03T17:10:28.585 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-05-03T17:10:28.781 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-05-03T17:10:28.827 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-05-03T17:10:28.905 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-05-03T17:10:28.994 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-05-03T17:10:29.000 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T17:10:29.133 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-05-03T17:10:29.144 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T17:10:29.177 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-05-03T17:10:29.209 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-05-03T17:10:29.237 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-05-03T17:10:29.673 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 123/156 2024-05-03T17:10:29.680 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 105/156 2024-05-03T17:10:29.702 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 123/156 2024-05-03T17:10:29.709 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 107/156 2024-05-03T17:10:29.709 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2024-05-03T17:10:29.709 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2024-05-03T17:10:29.709 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2024-05-03T17:10:29.709 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2024-05-03T17:10:29.709 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2024-05-03T17:10:29.709 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /root 2024-05-03T17:10:29.710 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2024-05-03T17:10:29.710 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:29.724 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 106/156 2024-05-03T17:10:29.734 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.e 124/156 2024-05-03T17:10:29.794 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7. 125/156 2024-05-03T17:10:29.817 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 106/156 2024-05-03T17:10:29.817 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-03T17:10:29.817 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-03T17:10:29.817 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:10:29.825 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 126/156 2024-05-03T17:10:29.842 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 107/156 2024-05-03T17:10:29.881 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 108/156 2024-05-03T17:10:29.897 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 108/156 2024-05-03T17:10:29.897 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:29.897 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-03T17:10:29.897 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:30.862 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 126/156 2024-05-03T17:10:30.901 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 127/156 2024-05-03T17:10:31.426 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 109/156 2024-05-03T17:10:31.453 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 109/156 2024-05-03T17:10:31.453 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:31.453 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-03T17:10:31.453 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-03T17:10:31.453 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-03T17:10:31.453 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:31.693 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 110/156 2024-05-03T17:10:31.717 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 110/156 2024-05-03T17:10:31.717 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:31.717 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-03T17:10:31.717 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-03T17:10:31.717 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-03T17:10:31.717 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:31.860 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 127/156 2024-05-03T17:10:31.885 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 128/156 2024-05-03T17:10:31.996 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 128/156 2024-05-03T17:10:32.155 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el 129/156 2024-05-03T17:10:32.185 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 111/156 2024-05-03T17:10:32.187 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 130/156 2024-05-03T17:10:32.209 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 111/156 2024-05-03T17:10:32.209 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:32.209 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-03T17:10:32.209 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-03T17:10:32.209 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-03T17:10:32.210 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:32.210 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 130/156 2024-05-03T17:10:32.210 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:32.210 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-03T17:10:32.210 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-03T17:10:32.211 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-03T17:10:32.211 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:32.254 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 131/156 2024-05-03T17:10:32.265 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 131/156 2024-05-03T17:10:32.324 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-05-03T17:10:32.363 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 132/156 2024-05-03T17:10:32.382 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-05-03T17:10:32.411 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 133/156 2024-05-03T17:10:32.447 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-05-03T17:10:32.641 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-05-03T17:10:32.695 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-05-03T17:10:32.776 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-05-03T17:10:32.866 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-05-03T17:10:33.019 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-05-03T17:10:33.064 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-05-03T17:10:33.096 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-05-03T17:10:33.122 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-05-03T17:10:33.336 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-05-03T17:10:33.373 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 135/156 2024-05-03T17:10:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 135/156 2024-05-03T17:10:33.396 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:33.396 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-03T17:10:33.396 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-03T17:10:33.396 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-03T17:10:33.396 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:33.441 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3450.g9a047 136/156 2024-05-03T17:10:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3450.g9a047 136/156 2024-05-03T17:10:33.463 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:33.463 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-03T17:10:33.463 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:33.508 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 137/156 2024-05-03T17:10:33.530 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 137/156 2024-05-03T17:10:33.530 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:33.531 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-03T17:10:33.531 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-03T17:10:33.531 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-03T17:10:33.531 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:33.535 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 123/156 2024-05-03T17:10:33.555 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 123/156 2024-05-03T17:10:33.587 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.e 124/156 2024-05-03T17:10:33.660 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7. 125/156 2024-05-03T17:10:33.686 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 126/156 2024-05-03T17:10:33.833 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 138/156 2024-05-03T17:10:33.856 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 138/156 2024-05-03T17:10:33.856 INFO:teuthology.orchestra.run.smithi082.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:33.856 INFO:teuthology.orchestra.run.smithi082.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-03T17:10:33.856 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-03T17:10:33.857 INFO:teuthology.orchestra.run.smithi082.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-03T17:10:33.857 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:34.783 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 126/156 2024-05-03T17:10:34.821 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 127/156 2024-05-03T17:10:35.842 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 127/156 2024-05-03T17:10:35.871 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 128/156 2024-05-03T17:10:35.989 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 128/156 2024-05-03T17:10:36.157 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el 129/156 2024-05-03T17:10:36.198 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 130/156 2024-05-03T17:10:36.222 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 130/156 2024-05-03T17:10:36.223 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:36.223 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-03T17:10:36.223 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-03T17:10:36.223 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-03T17:10:36.223 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:36.274 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 131/156 2024-05-03T17:10:36.285 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 131/156 2024-05-03T17:10:36.387 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 132/156 2024-05-03T17:10:36.444 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 133/156 2024-05-03T17:10:37.424 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-05-03T17:10:37.451 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 135/156 2024-05-03T17:10:37.476 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 135/156 2024-05-03T17:10:37.476 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:37.476 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-03T17:10:37.476 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-03T17:10:37.476 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-03T17:10:37.477 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:37.519 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3450.g9a047 136/156 2024-05-03T17:10:37.542 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3450.g9a047 136/156 2024-05-03T17:10:37.542 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:37.542 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-03T17:10:37.542 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:37.603 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 137/156 2024-05-03T17:10:37.627 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 137/156 2024-05-03T17:10:37.627 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:37.627 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-03T17:10:37.627 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-03T17:10:37.627 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-03T17:10:37.627 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:37.953 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 138/156 2024-05-03T17:10:37.977 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 138/156 2024-05-03T17:10:37.977 INFO:teuthology.orchestra.run.smithi067.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:37.977 INFO:teuthology.orchestra.run.smithi067.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-03T17:10:37.977 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-03T17:10:37.977 INFO:teuthology.orchestra.run.smithi067.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-03T17:10:37.977 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:38.453 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 139/156 2024-05-03T17:10:38.494 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 140/156 2024-05-03T17:10:38.535 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 141/156 2024-05-03T17:10:38.567 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-05-03T17:10:38.594 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-05-03T17:10:38.778 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 144/156 2024-05-03T17:10:38.824 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-05-03T17:10:38.854 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_ 146/156 2024-05-03T17:10:39.425 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T17:10:39.459 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T17:10:39.480 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T17:10:40.309 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 148/156 2024-05-03T17:10:40.397 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-05-03T17:10:40.438 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-05-03T17:10:40.518 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-05-03T17:10:40.551 INFO:teuthology.orchestra.run.smithi082.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-05-03T17:10:40.551 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-05-03T17:10:40.644 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-05-03T17:10:40.677 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-05-03T17:10:40.677 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-05-03T17:10:40.757 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-05-03T17:10:40.833 INFO:teuthology.orchestra.run.smithi082.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T17:10:41.421 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 107/156 2024-05-03T17:10:41.421 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2024-05-03T17:10:41.421 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2024-05-03T17:10:41.421 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2024-05-03T17:10:41.421 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2024-05-03T17:10:41.421 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2024-05-03T17:10:41.421 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /root 2024-05-03T17:10:41.421 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2024-05-03T17:10:41.422 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:10:41.605 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 108/156 2024-05-03T17:10:41.621 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 108/156 2024-05-03T17:10:41.621 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:41.621 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-03T17:10:41.621 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:10:42.832 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 139/156 2024-05-03T17:10:42.874 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 140/156 2024-05-03T17:10:42.915 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 141/156 2024-05-03T17:10:42.946 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-05-03T17:10:42.974 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-05-03T17:10:43.075 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 109/156 2024-05-03T17:10:43.101 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 109/156 2024-05-03T17:10:43.101 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:43.101 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-03T17:10:43.101 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-03T17:10:43.101 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-03T17:10:43.101 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:10:43.166 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 144/156 2024-05-03T17:10:43.211 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-05-03T17:10:43.233 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_ 146/156 2024-05-03T17:10:43.325 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 110/156 2024-05-03T17:10:43.347 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 110/156 2024-05-03T17:10:43.347 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:43.348 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-03T17:10:43.348 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-03T17:10:43.348 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-03T17:10:43.348 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:10:43.793 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 111/156 2024-05-03T17:10:43.816 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 111/156 2024-05-03T17:10:43.816 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:43.816 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-03T17:10:43.816 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-03T17:10:43.816 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-03T17:10:43.816 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:10:43.874 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T17:10:43.914 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T17:10:43.914 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-05-03T17:10:43.935 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T17:10:43.962 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-05-03T17:10:44.018 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-05-03T17:10:44.201 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-05-03T17:10:44.243 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-05-03T17:10:44.312 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-05-03T17:10:44.400 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-05-03T17:10:44.537 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-05-03T17:10:44.579 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-05-03T17:10:44.611 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-05-03T17:10:44.639 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-05-03T17:10:44.833 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 148/156 2024-05-03T17:10:44.919 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-05-03T17:10:44.961 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-05-03T17:10:45.053 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-05-03T17:10:45.065 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 123/156 2024-05-03T17:10:45.090 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-05-03T17:10:45.090 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-05-03T17:10:45.095 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 123/156 2024-05-03T17:10:45.127 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.e 124/156 2024-05-03T17:10:45.182 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-05-03T17:10:45.196 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7. 125/156 2024-05-03T17:10:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-05-03T17:10:45.215 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-05-03T17:10:45.235 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 126/156 2024-05-03T17:10:45.306 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-05-03T17:10:45.375 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T17:10:46.261 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 126/156 2024-05-03T17:10:46.295 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 127/156 2024-05-03T17:10:46.490 INFO:teuthology.orchestra.run.smithi082.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T17:10:46.490 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 1/156 2024-05-03T17:10:46.490 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 2/156 2024-05-03T17:10:46.490 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 3/156 2024-05-03T17:10:46.490 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 4/156 2024-05-03T17:10:46.490 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3450.g9a047 5/156 2024-05-03T17:10:46.490 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 6/156 2024-05-03T17:10:46.490 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 7/156 2024-05-03T17:10:46.490 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T17:10:46.490 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T17:10:46.490 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 10/156 2024-05-03T17:10:46.490 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 11/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 12/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 14/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_ 15/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 16/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 17/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_6 18/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 19/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 20/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el 21/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9. 22/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_6 23/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 24/156 2024-05-03T17:10:46.491 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 25/156 2024-05-03T17:10:46.492 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 26/156 2024-05-03T17:10:46.492 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 27/156 2024-05-03T17:10:46.492 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 28/156 2024-05-03T17:10:46.492 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 29/156 2024-05-03T17:10:46.503 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7. 30/156 2024-05-03T17:10:46.503 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 31/156 2024-05-03T17:10:46.503 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 32/156 2024-05-03T17:10:46.503 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 33/156 2024-05-03T17:10:46.503 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el 34/156 2024-05-03T17:10:46.503 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 35/156 2024-05-03T17:10:46.503 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.e 36/156 2024-05-03T17:10:46.503 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 37/156 2024-05-03T17:10:46.503 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 38/156 2024-05-03T17:10:46.503 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 39/156 2024-05-03T17:10:46.503 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-05-03T17:10:46.503 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-05-03T17:10:46.504 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-05-03T17:10:46.505 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-05-03T17:10:46.505 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-05-03T17:10:46.505 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-05-03T17:10:46.505 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-05-03T17:10:46.505 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-05-03T17:10:46.505 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-05-03T17:10:46.505 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-05-03T17:10:46.505 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-05-03T17:10:46.505 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-05-03T17:10:46.505 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-05-03T17:10:46.505 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-05-03T17:10:46.505 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-05-03T17:10:46.506 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-05-03T17:10:46.507 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-05-03T17:10:46.508 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-05-03T17:10:46.509 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-05-03T17:10:46.510 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-05-03T17:10:46.511 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-05-03T17:10:46.511 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 149/156 2024-05-03T17:10:46.511 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-05-03T17:10:46.511 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 151/156 2024-05-03T17:10:46.511 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-05-03T17:10:46.511 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-05-03T17:10:46.511 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-05-03T17:10:46.511 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-05-03T17:10:47.250 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 127/156 2024-05-03T17:10:47.279 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 128/156 2024-05-03T17:10:47.388 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 128/156 2024-05-03T17:10:47.547 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el 129/156 2024-05-03T17:10:47.581 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 130/156 2024-05-03T17:10:47.604 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 130/156 2024-05-03T17:10:47.604 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:47.604 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-03T17:10:47.604 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-03T17:10:47.604 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-03T17:10:47.604 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:10:47.656 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 131/156 2024-05-03T17:10:47.668 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 131/156 2024-05-03T17:10:47.765 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 132/156 2024-05-03T17:10:47.821 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 133/156 2024-05-03T17:10:48.619 INFO:teuthology.orchestra.run.smithi082.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T17:10:48.619 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:48.619 INFO:teuthology.orchestra.run.smithi082.stdout:Upgraded: 2024-05-03T17:10:48.619 INFO:teuthology.orchestra.run.smithi082.stdout: librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.619 INFO:teuthology.orchestra.run.smithi082.stdout: librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.619 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-03T17:10:48.619 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-03T17:10:48.619 INFO:teuthology.orchestra.run.smithi082.stdout:Installed: 2024-05-03T17:10:48.619 INFO:teuthology.orchestra.run.smithi082.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-03T17:10:48.619 INFO:teuthology.orchestra.run.smithi082.stdout: automake-1.16.2-8.el9.noarch 2024-05-03T17:10:48.619 INFO:teuthology.orchestra.run.smithi082.stdout: bison-3.7.4-5.el9.x86_64 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-immutable-object-cache-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.620 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: flex-2.6.4-9.el9.x86_64 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-03T17:10:48.621 INFO:teuthology.orchestra.run.smithi082.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.622 INFO:teuthology.orchestra.run.smithi082.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: lua-5.4.4-4.el9.x86_64 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-13.14-1.el9.x86_64 2024-05-03T17:10:48.623 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-03T17:10:48.624 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-03T17:10:48.625 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-ply-3.11-14.el9.noarch 2024-05-03T17:10:48.626 INFO:teuthology.orchestra.run.smithi082.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-03T17:10:48.627 INFO:teuthology.orchestra.run.smithi082.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-03T17:10:48.628 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.629 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.629 INFO:teuthology.orchestra.run.smithi082.stdout: rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:48.629 INFO:teuthology.orchestra.run.smithi082.stdout: re2-1:20211101-3.el9.x86_64 2024-05-03T17:10:48.629 INFO:teuthology.orchestra.run.smithi082.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-03T17:10:48.629 INFO:teuthology.orchestra.run.smithi082.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-03T17:10:48.629 INFO:teuthology.orchestra.run.smithi082.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-03T17:10:48.629 INFO:teuthology.orchestra.run.smithi082.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-03T17:10:48.629 INFO:teuthology.orchestra.run.smithi082.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-03T17:10:48.629 INFO:teuthology.orchestra.run.smithi082.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-03T17:10:48.629 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:10:48.630 INFO:teuthology.orchestra.run.smithi082.stdout:Complete! 2024-05-03T17:10:48.739 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-05-03T17:10:48.767 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 135/156 2024-05-03T17:10:48.789 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 135/156 2024-05-03T17:10:48.789 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:48.789 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-03T17:10:48.790 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-03T17:10:48.790 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-03T17:10:48.790 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:10:48.834 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3450.g9a047 136/156 2024-05-03T17:10:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3450.g9a047 136/156 2024-05-03T17:10:48.856 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:48.856 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-03T17:10:48.856 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:10:48.902 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 137/156 2024-05-03T17:10:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 137/156 2024-05-03T17:10:48.924 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:48.924 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-03T17:10:48.924 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-03T17:10:48.924 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-03T17:10:48.924 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:10:48.998 DEBUG:teuthology.parallel:result is None 2024-05-03T17:10:49.244 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 138/156 2024-05-03T17:10:49.266 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 138/156 2024-05-03T17:10:49.267 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-03T17:10:49.267 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-03T17:10:49.267 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-03T17:10:49.267 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-03T17:10:49.267 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:10:51.245 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T17:10:51.245 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 1/156 2024-05-03T17:10:51.245 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 2/156 2024-05-03T17:10:51.245 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 3/156 2024-05-03T17:10:51.245 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 4/156 2024-05-03T17:10:51.245 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3450.g9a047 5/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 6/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 7/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 10/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 11/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 12/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 14/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_ 15/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 16/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 17/156 2024-05-03T17:10:51.246 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_6 18/156 2024-05-03T17:10:51.247 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 19/156 2024-05-03T17:10:51.247 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 20/156 2024-05-03T17:10:51.247 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el 21/156 2024-05-03T17:10:51.247 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9. 22/156 2024-05-03T17:10:51.247 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_6 23/156 2024-05-03T17:10:51.247 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 24/156 2024-05-03T17:10:51.247 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 25/156 2024-05-03T17:10:51.247 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 26/156 2024-05-03T17:10:51.260 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 27/156 2024-05-03T17:10:51.260 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 28/156 2024-05-03T17:10:51.260 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 29/156 2024-05-03T17:10:51.260 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7. 30/156 2024-05-03T17:10:51.260 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 31/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 32/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 33/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el 34/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 35/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.e 36/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 37/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 38/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 39/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-05-03T17:10:51.261 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-05-03T17:10:51.262 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-05-03T17:10:51.263 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-05-03T17:10:51.264 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-05-03T17:10:51.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-05-03T17:10:51.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-05-03T17:10:51.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-05-03T17:10:51.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-05-03T17:10:51.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-05-03T17:10:51.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-05-03T17:10:51.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-05-03T17:10:51.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-05-03T17:10:51.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-05-03T17:10:51.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-05-03T17:10:51.265 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-05-03T17:10:51.266 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-05-03T17:10:51.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-05-03T17:10:51.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-05-03T17:10:51.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-05-03T17:10:51.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-05-03T17:10:51.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-05-03T17:10:51.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-05-03T17:10:51.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-05-03T17:10:51.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-05-03T17:10:51.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-05-03T17:10:51.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-05-03T17:10:51.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-05-03T17:10:51.267 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-05-03T17:10:51.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-05-03T17:10:51.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-05-03T17:10:51.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-05-03T17:10:51.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-05-03T17:10:51.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-05-03T17:10:51.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-05-03T17:10:51.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-05-03T17:10:51.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-05-03T17:10:51.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-05-03T17:10:51.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-05-03T17:10:51.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-05-03T17:10:51.268 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-05-03T17:10:51.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-05-03T17:10:51.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-05-03T17:10:51.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-05-03T17:10:51.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-05-03T17:10:51.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-05-03T17:10:51.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-05-03T17:10:51.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-05-03T17:10:51.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-05-03T17:10:51.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-05-03T17:10:51.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-05-03T17:10:51.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-05-03T17:10:51.269 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-05-03T17:10:51.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-05-03T17:10:51.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 149/156 2024-05-03T17:10:51.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-05-03T17:10:51.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 151/156 2024-05-03T17:10:51.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-05-03T17:10:51.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-05-03T17:10:51.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-05-03T17:10:51.270 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-05-03T17:10:53.309 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T17:10:53.309 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:53.309 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2024-05-03T17:10:53.309 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.309 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: automake-1.16.2-8.el9.noarch 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: bison-3.7.4-5.el9.x86_64 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:53.310 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:53.311 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: flex-2.6.4-9.el9.x86_64 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-03T17:10:53.312 INFO:teuthology.orchestra.run.smithi067.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.313 INFO:teuthology.orchestra.run.smithi067.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: lua-5.4.4-4.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-13.14-1.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-03T17:10:53.314 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-03T17:10:53.315 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-03T17:10:53.316 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ply-3.11-14.el9.noarch 2024-05-03T17:10:53.317 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.318 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-03T17:10:53.319 INFO:teuthology.orchestra.run.smithi067.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-03T17:10:53.320 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.320 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.320 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:10:53.320 INFO:teuthology.orchestra.run.smithi067.stdout: re2-1:20211101-3.el9.x86_64 2024-05-03T17:10:53.320 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-03T17:10:53.320 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-03T17:10:53.320 INFO:teuthology.orchestra.run.smithi067.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-03T17:10:53.320 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-03T17:10:53.320 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-03T17:10:53.320 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-03T17:10:53.321 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:10:53.321 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2024-05-03T17:10:53.724 DEBUG:teuthology.parallel:result is None 2024-05-03T17:10:53.881 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 139/156 2024-05-03T17:10:53.923 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 140/156 2024-05-03T17:10:53.963 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 141/156 2024-05-03T17:10:53.995 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-05-03T17:10:54.022 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-05-03T17:10:54.203 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 144/156 2024-05-03T17:10:54.243 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-05-03T17:10:54.281 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_ 146/156 2024-05-03T17:10:54.842 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T17:10:54.887 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T17:10:54.907 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-05-03T17:10:55.726 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 148/156 2024-05-03T17:10:55.816 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-05-03T17:10:55.857 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-05-03T17:10:55.946 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-05-03T17:10:55.978 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-05-03T17:10:55.979 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-05-03T17:10:56.062 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-05-03T17:10:56.087 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-05-03T17:10:56.087 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-05-03T17:10:56.177 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-05-03T17:10:56.252 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T17:11:02.070 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T17:11:02.070 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 1/156 2024-05-03T17:11:02.070 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 2/156 2024-05-03T17:11:02.070 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 3/156 2024-05-03T17:11:02.070 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 4/156 2024-05-03T17:11:02.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3450.g9a047 5/156 2024-05-03T17:11:02.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 6/156 2024-05-03T17:11:02.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 7/156 2024-05-03T17:11:02.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 8/156 2024-05-03T17:11:02.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 9/156 2024-05-03T17:11:02.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 10/156 2024-05-03T17:11:02.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 11/156 2024-05-03T17:11:02.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 12/156 2024-05-03T17:11:02.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 13/156 2024-05-03T17:11:02.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 14/156 2024-05-03T17:11:02.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_ 15/156 2024-05-03T17:11:02.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 16/156 2024-05-03T17:11:02.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 17/156 2024-05-03T17:11:02.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_6 18/156 2024-05-03T17:11:02.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86 19/156 2024-05-03T17:11:02.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 20/156 2024-05-03T17:11:02.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el 21/156 2024-05-03T17:11:02.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9. 22/156 2024-05-03T17:11:02.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_6 23/156 2024-05-03T17:11:02.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 24/156 2024-05-03T17:11:02.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 25/156 2024-05-03T17:11:02.086 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 26/156 2024-05-03T17:11:02.086 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 27/156 2024-05-03T17:11:02.086 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 28/156 2024-05-03T17:11:02.086 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 29/156 2024-05-03T17:11:02.086 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7. 30/156 2024-05-03T17:11:02.086 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noa 31/156 2024-05-03T17:11:02.086 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.n 32/156 2024-05-03T17:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a0 33/156 2024-05-03T17:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el 34/156 2024-05-03T17:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 35/156 2024-05-03T17:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.e 36/156 2024-05-03T17:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 37/156 2024-05-03T17:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 38/156 2024-05-03T17:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 39/156 2024-05-03T17:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-05-03T17:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-05-03T17:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-05-03T17:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-05-03T17:11:02.087 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-05-03T17:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-05-03T17:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-05-03T17:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-05-03T17:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-05-03T17:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-05-03T17:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-05-03T17:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-05-03T17:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-05-03T17:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-05-03T17:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-05-03T17:11:02.088 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-05-03T17:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-05-03T17:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-05-03T17:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-05-03T17:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-05-03T17:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-05-03T17:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-05-03T17:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-05-03T17:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-05-03T17:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-05-03T17:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-05-03T17:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-05-03T17:11:02.089 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-05-03T17:11:02.090 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-05-03T17:11:02.090 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-05-03T17:11:02.090 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-05-03T17:11:02.090 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-05-03T17:11:02.090 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-05-03T17:11:02.090 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-05-03T17:11:02.090 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-05-03T17:11:02.090 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-05-03T17:11:02.090 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-05-03T17:11:02.090 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-05-03T17:11:02.090 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-05-03T17:11:02.091 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-05-03T17:11:02.092 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-05-03T17:11:02.092 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-05-03T17:11:02.092 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-05-03T17:11:02.092 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-05-03T17:11:02.092 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-05-03T17:11:02.092 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-05-03T17:11:02.092 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-05-03T17:11:02.092 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-05-03T17:11:02.092 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-05-03T17:11:02.092 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-05-03T17:11:02.092 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-05-03T17:11:02.092 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-05-03T17:11:02.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-05-03T17:11:02.094 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-05-03T17:11:02.094 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-05-03T17:11:02.094 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-05-03T17:11:02.094 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-05-03T17:11:02.094 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-05-03T17:11:02.094 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-05-03T17:11:02.094 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-05-03T17:11:02.094 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-05-03T17:11:02.094 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-05-03T17:11:02.094 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-05-03T17:11:02.094 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-05-03T17:11:02.094 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-05-03T17:11:02.095 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-05-03T17:11:02.095 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-05-03T17:11:02.095 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-05-03T17:11:02.095 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-05-03T17:11:02.095 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-05-03T17:11:02.095 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-05-03T17:11:02.095 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-05-03T17:11:02.095 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-05-03T17:11:02.095 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-05-03T17:11:02.095 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-05-03T17:11:02.095 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-05-03T17:11:02.096 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-05-03T17:11:02.097 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-05-03T17:11:02.097 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-05-03T17:11:02.097 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-05-03T17:11:02.097 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-05-03T17:11:02.097 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-05-03T17:11:02.097 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-05-03T17:11:02.097 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-05-03T17:11:02.097 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-05-03T17:11:02.098 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 149/156 2024-05-03T17:11:02.098 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-05-03T17:11:02.098 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 151/156 2024-05-03T17:11:02.098 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-05-03T17:11:02.098 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-05-03T17:11:02.098 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-05-03T17:11:02.098 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-05-03T17:11:04.271 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-05-03T17:11:04.271 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:11:04.271 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2024-05-03T17:11:04.271 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.272 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.272 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-03T17:11:04.272 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-03T17:11:04.272 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-05-03T17:11:04.272 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-03T17:11:04.272 INFO:teuthology.orchestra.run.smithi070.stdout: automake-1.16.2-8.el9.noarch 2024-05-03T17:11:04.273 INFO:teuthology.orchestra.run.smithi070.stdout: bison-3.7.4-5.el9.x86_64 2024-05-03T17:11:04.273 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-03T17:11:04.273 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.273 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.273 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.273 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.273 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:11:04.274 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.274 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.274 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.274 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:11:04.274 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:11:04.274 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:11:04.275 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:11:04.275 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:11:04.275 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.275 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.275 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:11:04.275 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.276 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.276 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.276 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:11:04.276 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:11:04.276 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.276 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-shell-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.276 INFO:teuthology.orchestra.run.smithi070.stdout: cephfs-top-2:19.0.0-3450.g9a047fe7.el9.noarch 2024-05-03T17:11:04.277 INFO:teuthology.orchestra.run.smithi070.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-03T17:11:04.277 INFO:teuthology.orchestra.run.smithi070.stdout: flex-2.6.4-9.el9.x86_64 2024-05-03T17:11:04.277 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-03T17:11:04.277 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-03T17:11:04.277 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-03T17:11:04.277 INFO:teuthology.orchestra.run.smithi070.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-03T17:11:04.277 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-03T17:11:04.277 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-03T17:11:04.278 INFO:teuthology.orchestra.run.smithi070.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-03T17:11:04.278 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-03T17:11:04.278 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-03T17:11:04.278 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.278 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.278 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.279 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-03T17:11:04.279 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-03T17:11:04.279 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-03T17:11:04.279 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-03T17:11:04.279 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-03T17:11:04.279 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.279 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.280 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-03T17:11:04.280 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.280 INFO:teuthology.orchestra.run.smithi070.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-03T17:11:04.280 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-03T17:11:04.280 INFO:teuthology.orchestra.run.smithi070.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-03T17:11:04.280 INFO:teuthology.orchestra.run.smithi070.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-03T17:11:04.280 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-03T17:11:04.281 INFO:teuthology.orchestra.run.smithi070.stdout: lua-5.4.4-4.el9.x86_64 2024-05-03T17:11:04.281 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-03T17:11:04.281 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-03T17:11:04.281 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-03T17:11:04.281 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-03T17:11:04.281 INFO:teuthology.orchestra.run.smithi070.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-03T17:11:04.281 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-03T17:11:04.282 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-13.14-1.el9.x86_64 2024-05-03T17:11:04.282 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-03T17:11:04.282 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-03T17:11:04.282 INFO:teuthology.orchestra.run.smithi070.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-03T17:11:04.282 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-03T17:11:04.282 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-03T17:11:04.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-03T17:11:04.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-03T17:11:04.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-03T17:11:04.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-03T17:11:04.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-03T17:11:04.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.283 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.284 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.284 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-03T17:11:04.284 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-03T17:11:04.284 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-03T17:11:04.284 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-03T17:11:04.284 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-03T17:11:04.284 INFO:teuthology.orchestra.run.smithi070.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-03T17:11:04.285 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-03T17:11:04.285 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-03T17:11:04.285 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-03T17:11:04.285 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-03T17:11:04.285 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-03T17:11:04.285 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-03T17:11:04.285 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-03T17:11:04.286 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-03T17:11:04.286 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-03T17:11:04.286 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-03T17:11:04.286 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-03T17:11:04.286 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-03T17:11:04.286 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-03T17:11:04.287 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-03T17:11:04.287 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-03T17:11:04.287 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-03T17:11:04.287 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-03T17:11:04.287 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-03T17:11:04.287 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-03T17:11:04.288 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-03T17:11:04.288 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-03T17:11:04.288 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-03T17:11:04.288 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-03T17:11:04.288 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-03T17:11:04.288 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply-3.11-14.el9.noarch 2024-05-03T17:11:04.288 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-03T17:11:04.289 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-03T17:11:04.289 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-03T17:11:04.289 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-03T17:11:04.289 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-03T17:11:04.289 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-03T17:11:04.289 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-03T17:11:04.289 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-03T17:11:04.290 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-03T17:11:04.290 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.290 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.290 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-03T17:11:04.290 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-03T17:11:04.290 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-03T17:11:04.291 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.291 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-03T17:11:04.291 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-03T17:11:04.291 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-03T17:11:04.291 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-03T17:11:04.291 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-03T17:11:04.291 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-03T17:11:04.292 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-03T17:11:04.292 INFO:teuthology.orchestra.run.smithi070.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-03T17:11:04.292 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-03T17:11:04.292 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-03T17:11:04.292 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-03T17:11:04.292 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-03T17:11:04.292 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-03T17:11:04.293 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-03T17:11:04.293 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-03T17:11:04.293 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.293 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.293 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:19.0.0-3450.g9a047fe7.el9.x86_64 2024-05-03T17:11:04.293 INFO:teuthology.orchestra.run.smithi070.stdout: re2-1:20211101-3.el9.x86_64 2024-05-03T17:11:04.293 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-03T17:11:04.294 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-03T17:11:04.294 INFO:teuthology.orchestra.run.smithi070.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-03T17:11:04.294 INFO:teuthology.orchestra.run.smithi070.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-03T17:11:04.294 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-03T17:11:04.294 INFO:teuthology.orchestra.run.smithi070.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-03T17:11:04.294 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-05-03T17:11:04.295 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-05-03T17:11:04.681 DEBUG:teuthology.parallel:result is None 2024-05-03T17:11:04.681 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T17:11:04.823 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-03T17:11:04.850 INFO:teuthology.orchestra.run.smithi067.stdout:19.0.0-3450.g9a047fe7.el9 2024-05-03T17:11:04.850 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3450.g9a047fe7.el9 2024-05-03T17:11:04.850 INFO:teuthology.task.install:The correct ceph version 19.0.0-3450.g9a047fe7 is installed. 2024-05-03T17:11:04.852 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T17:11:04.992 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-03T17:11:05.013 INFO:teuthology.orchestra.run.smithi070.stdout:19.0.0-3450.g9a047fe7.el9 2024-05-03T17:11:05.013 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3450.g9a047fe7.el9 2024-05-03T17:11:05.013 INFO:teuthology.task.install:The correct ceph version 19.0.0-3450.g9a047fe7 is installed. 2024-05-03T17:11:05.015 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T17:11:05.232 DEBUG:teuthology.orchestra.run.smithi082:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-03T17:11:05.258 INFO:teuthology.orchestra.run.smithi082.stdout:19.0.0-3450.g9a047fe7.el9 2024-05-03T17:11:05.258 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3450.g9a047fe7.el9 2024-05-03T17:11:05.258 INFO:teuthology.task.install:The correct ceph version 19.0.0-3450.g9a047fe7 is installed. 2024-05-03T17:11:05.260 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-03T17:11:05.260 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:11:05.260 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-03T17:11:05.292 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-05-03T17:11:05.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-03T17:11:05.318 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T17:11:05.318 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-03T17:11:05.348 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-03T17:11:05.349 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:11:05.349 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2024-05-03T17:11:05.379 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-03T17:11:05.447 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-05-03T17:11:05.448 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-05-03T17:11:05.474 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-03T17:11:05.538 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T17:11:05.538 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/daemon-helper 2024-05-03T17:11:05.566 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-03T17:11:05.635 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-03T17:11:05.635 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:11:05.635 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-03T17:11:05.665 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-03T17:11:05.733 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-05-03T17:11:05.734 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-03T17:11:05.757 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-03T17:11:05.822 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T17:11:05.823 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-03T17:11:05.846 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-03T17:11:05.911 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-03T17:11:05.911 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:11:05.911 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2024-05-03T17:11:05.940 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-03T17:11:06.008 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-05-03T17:11:06.008 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-05-03T17:11:06.033 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-03T17:11:06.098 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T17:11:06.098 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/usr/bin/stdin-killer 2024-05-03T17:11:06.122 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-03T17:11:06.185 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-03T17:11:06.279 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-05-03T17:11:06.279 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T17:11:06.279 INFO:tasks.cephadm:Cluster fsid is 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:11:06.280 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-03T17:11:06.280 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.67', 'mon.b': '172.21.15.70', 'mon.c': '172.21.15.82'} 2024-05-03T17:11:06.280 INFO:tasks.cephadm:First mon is mon.a on smithi067 2024-05-03T17:11:06.280 INFO:tasks.cephadm:First mgr is x 2024-05-03T17:11:06.280 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-03T17:11:06.280 DEBUG:teuthology.orchestra.run.smithi067:> sudo hostname $(hostname -s) 2024-05-03T17:11:06.308 DEBUG:teuthology.orchestra.run.smithi070:> sudo hostname $(hostname -s) 2024-05-03T17:11:06.332 DEBUG:teuthology.orchestra.run.smithi082:> sudo hostname $(hostname -s) 2024-05-03T17:11:06.356 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-03T17:11:06.356 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T17:11:06.502 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '9a047fe7599a549042d06a3fb5b6813a203eabea', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79034/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3450-g9a047fe7', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3450.g9a047fe7'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-02 00:36:37.664792', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-rishabh-testing-20240501.193033', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-05-03T17:11:06.639 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-rishabh-testing-20240501.193033, sha1 9a047fe7599a549042d06a3fb5b6813a203eabea from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T17:11:06.640 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/x86_64/flavors/default/cephadm 2024-05-03T17:11:06.640 DEBUG:teuthology.orchestra.run.smithi067:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-03T17:11:07.084 INFO:teuthology.orchestra.run.smithi067.stdout:-rw-r--r--. 1 ubuntu ubuntu 782467 May 3 17:11 /home/ubuntu/cephtest/cephadm 2024-05-03T17:11:07.085 DEBUG:teuthology.orchestra.run.smithi070:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-03T17:11:07.727 INFO:teuthology.orchestra.run.smithi070.stdout:-rw-r--r--. 1 ubuntu ubuntu 782467 May 3 17:11 /home/ubuntu/cephtest/cephadm 2024-05-03T17:11:07.727 DEBUG:teuthology.orchestra.run.smithi082:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-rishabh-testing-20240501.193033/9a047fe7599a549042d06a3fb5b6813a203eabea/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-03T17:11:08.175 INFO:teuthology.orchestra.run.smithi082.stdout:-rw-r--r--. 1 ubuntu ubuntu 782467 May 3 17:11 /home/ubuntu/cephtest/cephadm 2024-05-03T17:11:08.175 DEBUG:teuthology.orchestra.run.smithi067:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-03T17:11:08.194 DEBUG:teuthology.orchestra.run.smithi070:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-03T17:11:08.213 DEBUG:teuthology.orchestra.run.smithi082:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-03T17:11:08.247 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea on all hosts... 2024-05-03T17:11:08.248 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea pull 2024-05-03T17:11:08.250 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea pull 2024-05-03T17:11:08.256 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea pull 2024-05-03T17:11:09.777 INFO:teuthology.orchestra.run.smithi082.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea... 2024-05-03T17:11:09.959 INFO:teuthology.orchestra.run.smithi067.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea... 2024-05-03T17:11:09.992 INFO:teuthology.orchestra.run.smithi070.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea... 2024-05-03T17:11:48.728 INFO:teuthology.orchestra.run.smithi070.stdout:{ 2024-05-03T17:11:48.728 INFO:teuthology.orchestra.run.smithi070.stdout: "ceph_version": "ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev)", 2024-05-03T17:11:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "image_id": "ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012", 2024-05-03T17:11:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "repo_digests": [ 2024-05-03T17:11:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd" 2024-05-03T17:11:48.729 INFO:teuthology.orchestra.run.smithi070.stdout: ] 2024-05-03T17:11:48.729 INFO:teuthology.orchestra.run.smithi070.stdout:} 2024-05-03T17:11:50.358 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T17:11:50.359 INFO:teuthology.orchestra.run.smithi067.stdout: "ceph_version": "ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev)", 2024-05-03T17:11:50.359 INFO:teuthology.orchestra.run.smithi067.stdout: "image_id": "ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012", 2024-05-03T17:11:50.359 INFO:teuthology.orchestra.run.smithi067.stdout: "repo_digests": [ 2024-05-03T17:11:50.359 INFO:teuthology.orchestra.run.smithi067.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd" 2024-05-03T17:11:50.359 INFO:teuthology.orchestra.run.smithi067.stdout: ] 2024-05-03T17:11:50.359 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T17:11:50.892 INFO:teuthology.orchestra.run.smithi082.stdout:{ 2024-05-03T17:11:50.892 INFO:teuthology.orchestra.run.smithi082.stdout: "ceph_version": "ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev)", 2024-05-03T17:11:50.892 INFO:teuthology.orchestra.run.smithi082.stdout: "image_id": "ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012", 2024-05-03T17:11:50.892 INFO:teuthology.orchestra.run.smithi082.stdout: "repo_digests": [ 2024-05-03T17:11:50.892 INFO:teuthology.orchestra.run.smithi082.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd" 2024-05-03T17:11:50.893 INFO:teuthology.orchestra.run.smithi082.stdout: ] 2024-05-03T17:11:50.893 INFO:teuthology.orchestra.run.smithi082.stdout:} 2024-05-03T17:11:50.916 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph 2024-05-03T17:11:50.947 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph 2024-05-03T17:11:50.978 DEBUG:teuthology.orchestra.run.smithi082:> sudo mkdir -p /etc/ceph 2024-05-03T17:11:51.004 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /etc/ceph 2024-05-03T17:11:51.031 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /etc/ceph 2024-05-03T17:11:51.058 DEBUG:teuthology.orchestra.run.smithi082:> sudo chmod 777 /etc/ceph 2024-05-03T17:11:51.082 INFO:tasks.cephadm:Writing seed config... 2024-05-03T17:11:51.083 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-03T17:11:51.083 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-03T17:11:51.083 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-03T17:11:51.083 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-03T17:11:51.083 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-03T17:11:51.083 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-03T17:11:51.083 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-03T17:11:51.083 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-03T17:11:51.084 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-03T17:11:51.085 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-03T17:11:51.086 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-03T17:11:51.086 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-03T17:11:51.086 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-03T17:11:51.086 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-03T17:11:51.086 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-03T17:11:51.086 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-03T17:11:51.086 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-03T17:11:51.086 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-03T17:11:51.086 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-03T17:11:51.086 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-05-03T17:11:51.087 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:11:51.087 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-03T17:11:51.104 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 20c8e22e-0970-11ef-bc97-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-03T17:11:51.105 DEBUG:teuthology.orchestra.run.smithi067:mon.a> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.a.service 2024-05-03T17:11:51.147 DEBUG:teuthology.orchestra.run.smithi067:mgr.x> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.x.service 2024-05-03T17:11:51.189 INFO:tasks.cephadm:Bootstrapping... 2024-05-03T17:11:51.189 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea -v bootstrap --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.67 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-03T17:11:51.391 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2024-05-03T17:11:51.391 INFO:teuthology.orchestra.run.smithi067.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea', '-v', 'bootstrap', '--fsid', '20c8e22e-0970-11ef-bc97-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.67', '--skip-admin-label'] 2024-05-03T17:11:51.392 INFO:teuthology.orchestra.run.smithi067.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-03T17:11:51.392 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying podman|docker is present... 2024-05-03T17:11:51.419 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-03T17:11:51.420 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying lvm2 is present... 2024-05-03T17:11:51.420 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying time synchronization is in place... 2024-05-03T17:11:51.428 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-03T17:11:51.428 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-03T17:11:51.434 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-03T17:11:51.434 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-05-03T17:11:51.440 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout enabled 2024-05-03T17:11:51.447 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout active 2024-05-03T17:11:51.447 INFO:teuthology.orchestra.run.smithi067.stdout:Unit chronyd.service is enabled and running 2024-05-03T17:11:51.447 INFO:teuthology.orchestra.run.smithi067.stdout:Repeating the final host check... 2024-05-03T17:11:51.471 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-03T17:11:51.471 INFO:teuthology.orchestra.run.smithi067.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-03T17:11:51.471 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl is present 2024-05-03T17:11:51.471 INFO:teuthology.orchestra.run.smithi067.stdout:lvcreate is present 2024-05-03T17:11:51.478 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-03T17:11:51.478 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-03T17:11:51.484 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-03T17:11:51.484 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-05-03T17:11:51.490 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout enabled 2024-05-03T17:11:51.497 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout active 2024-05-03T17:11:51.497 INFO:teuthology.orchestra.run.smithi067.stdout:Unit chronyd.service is enabled and running 2024-05-03T17:11:51.497 INFO:teuthology.orchestra.run.smithi067.stdout:Host looks OK 2024-05-03T17:11:51.497 INFO:teuthology.orchestra.run.smithi067.stdout:Cluster fsid: 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:11:51.497 INFO:teuthology.orchestra.run.smithi067.stdout:Acquiring lock 140336051273008 on /run/cephadm/20c8e22e-0970-11ef-bc97-c7b262605968.lock 2024-05-03T17:11:51.497 INFO:teuthology.orchestra.run.smithi067.stdout:Lock 140336051273008 acquired on /run/cephadm/20c8e22e-0970-11ef-bc97-c7b262605968.lock 2024-05-03T17:11:51.497 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying IP 172.21.15.67 port 3300 ... 2024-05-03T17:11:51.498 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying IP 172.21.15.67 port 6789 ... 2024-05-03T17:11:51.498 INFO:teuthology.orchestra.run.smithi067.stdout:Base mon IP(s) is [172.21.15.67:3300, 172.21.15.67:6789], mon addrv is [v2:172.21.15.67:3300,v1:172.21.15.67:6789] 2024-05-03T17:11:51.501 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.67 metric 100 2024-05-03T17:11:51.501 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.67 metric 100 2024-05-03T17:11:51.503 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-03T17:11:51.504 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-05-03T17:11:51.504 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-05-03T17:11:51.506 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-03T17:11:51.506 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-03T17:11:51.506 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-03T17:11:51.506 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-03T17:11:51.506 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout inet6 fe80::2e0f:da1f:ecf7:40f6/64 scope link noprefixroute 2024-05-03T17:11:51.506 INFO:teuthology.orchestra.run.smithi067.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-03T17:11:51.507 INFO:teuthology.orchestra.run.smithi067.stdout:Mon IP `172.21.15.67` is in CIDR network `172.21.0.0/20` 2024-05-03T17:11:51.507 INFO:teuthology.orchestra.run.smithi067.stdout:Mon IP `172.21.15.67` is in CIDR network `172.21.0.0/20` 2024-05-03T17:11:51.507 INFO:teuthology.orchestra.run.smithi067.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-03T17:11:51.508 INFO:teuthology.orchestra.run.smithi067.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-03T17:11:51.508 INFO:teuthology.orchestra.run.smithi067.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea... 2024-05-03T17:11:51.970 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stdout ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 2024-05-03T17:11:51.970 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea... 2024-05-03T17:11:51.970 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Getting image source signatures 2024-05-03T17:11:51.970 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Copying blob sha256:e2f2fd2adca92139da4b4b907fa90a6775d3eb4eddaa977c42ef37c66eea70bd 2024-05-03T17:11:51.970 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Copying blob sha256:8173fd91c96404ae0424bb08037c57749051dcbdfaeee9354a65e201dad7b0e2 2024-05-03T17:11:51.970 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Copying config sha256:ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 2024-05-03T17:11:51.970 INFO:teuthology.orchestra.run.smithi067.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-03T17:11:52.999 INFO:teuthology.orchestra.run.smithi067.stdout:ceph: stdout ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev) 2024-05-03T17:11:52.999 INFO:teuthology.orchestra.run.smithi067.stdout:Ceph version: ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev) 2024-05-03T17:11:52.999 INFO:teuthology.orchestra.run.smithi067.stdout:Extracting ceph user uid/gid from container image... 2024-05-03T17:12:01.475 INFO:teuthology.orchestra.run.smithi067.stdout:stat: stdout 167 167 2024-05-03T17:12:01.475 INFO:teuthology.orchestra.run.smithi067.stdout:Creating initial keys... 2024-05-03T17:12:05.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-authtool: stdout AQDjGjVmYiy/OBAAnDo3Qoqv+33pFklUbi9zXw== 2024-05-03T17:12:06.205 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-authtool: stdout AQDlGjVmSrBUJRAArJNW1W9AhX9f94D4ODIjyw== 2024-05-03T17:12:07.660 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph-authtool: stdout AQDmGjVm7JUlHhAA4GAvBZWR0IK5d2tWF0XeLg== 2024-05-03T17:12:07.660 INFO:teuthology.orchestra.run.smithi067.stdout:Creating initial monmap... 2024-05-03T17:12:08.558 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-03T17:12:08.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-03T17:12:08.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:12:08.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-03T17:12:08.559 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool for a [v2:172.21.15.67:3300,v1:172.21.15.67:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-03T17:12:08.559 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = quincy 2024-05-03T17:12:08.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: set fsid to 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:12:08.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-03T17:12:08.559 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:12:08.559 INFO:teuthology.orchestra.run.smithi067.stdout:Creating mon... 2024-05-03T17:12:09.463 INFO:teuthology.orchestra.run.smithi067.stdout:create mon.a on 2024-05-03T17:12:09.669 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-03T17:12:09.812 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-03T17:12:09.994 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-20c8e22e-0970-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-20c8e22e-0970-11ef-bc97-c7b262605968.target. 2024-05-03T17:12:09.994 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-20c8e22e-0970-11ef-bc97-c7b262605968.target → /etc/systemd/system/ceph-20c8e22e-0970-11ef-bc97-c7b262605968.target. 2024-05-03T17:12:10.702 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.a 2024-05-03T17:12:10.702 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to reset failed state of unit ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.a.service: Unit ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.a.service not loaded. 2024-05-03T17:12:10.978 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-20c8e22e-0970-11ef-bc97-c7b262605968.target.wants/ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.a.service → /etc/systemd/system/ceph-20c8e22e-0970-11ef-bc97-c7b262605968@.service. 2024-05-03T17:12:11.744 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-03T17:12:11.744 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-05-03T17:12:11.751 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-03T17:12:11.751 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-05-03T17:12:11.751 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-05-03T17:12:11.751 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to enable service . firewalld.service is not available 2024-05-03T17:12:11.752 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mon to start... 2024-05-03T17:12:11.752 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mon... 2024-05-03T17:12:11.863 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:11 smithi067 systemd[1]: Started Ceph mon.a for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:12:12.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout cluster: 2024-05-03T17:12:12.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout id: 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:12:12.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-03T17:12:12.969 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout services: 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.563484s) 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout data: 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout pgs: 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.191+0000 7fa19e38c700 1 Processor -- start 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.191+0000 7fa19e38c700 1 -- start start 2024-05-03T17:12:12.970 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.191+0000 7fa19e38c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 0x7fa198109870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.191+0000 7fa19e38c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa198109db0 con 0x7fa198109450 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.192+0000 7fa197fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 0x7fa198109870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.192+0000 7fa197fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 0x7fa198109870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35054/0 (socket says 172.21.15.67:35054) 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.192+0000 7fa197fff700 1 -- 172.21.15.67:0/768908086 learned_addr learned my addr 172.21.15.67:0/768908086 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.192+0000 7fa197fff700 1 -- 172.21.15.67:0/768908086 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa198107a30 con 0x7fa198109450 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.192+0000 7fa197fff700 1 --2- 172.21.15.67:0/768908086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 0x7fa198109870 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa1800098a0 tx=0x7fa180009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f44b4a89279553 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.192+0000 7fa196ffd700 1 -- 172.21.15.67:0/768908086 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa180004160 con 0x7fa198109450 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.192+0000 7fa196ffd700 1 -- 172.21.15.67:0/768908086 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa180004300 con 0x7fa198109450 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.192+0000 7fa196ffd700 1 -- 172.21.15.67:0/768908086 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa1800036a0 con 0x7fa198109450 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.193+0000 7fa19e38c700 1 -- 172.21.15.67:0/768908086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 msgr2=0x7fa198109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.193+0000 7fa19e38c700 1 --2- 172.21.15.67:0/768908086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 0x7fa198109870 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fa1800098a0 tx=0x7fa180009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.193+0000 7fa19e38c700 1 -- 172.21.15.67:0/768908086 shutdown_connections 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.193+0000 7fa19e38c700 1 --2- 172.21.15.67:0/768908086 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 0x7fa198109870 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:12.971 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.193+0000 7fa19e38c700 1 -- 172.21.15.67:0/768908086 >> 172.21.15.67:0/768908086 conn(0x7fa1981033f0 msgr2=0x7fa198105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.193+0000 7fa19e38c700 1 -- 172.21.15.67:0/768908086 shutdown_connections 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.193+0000 7fa19e38c700 1 -- 172.21.15.67:0/768908086 wait complete. 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.194+0000 7fa19e38c700 1 Processor -- start 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.194+0000 7fa19e38c700 1 -- start start 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.194+0000 7fa19e38c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 0x7fa19807d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.194+0000 7fa19e38c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa198107f00 con 0x7fa198109450 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.194+0000 7fa197fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 0x7fa19807d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.194+0000 7fa197fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 0x7fa19807d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35066/0 (socket says 172.21.15.67:35066) 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.194+0000 7fa197fff700 1 -- 172.21.15.67:0/1034919213 learned_addr learned my addr 172.21.15.67:0/1034919213 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.194+0000 7fa197fff700 1 -- 172.21.15.67:0/1034919213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa19819d210 con 0x7fa198109450 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.194+0000 7fa197fff700 1 --2- 172.21.15.67:0/1034919213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 0x7fa19807d300 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa180000c00 tx=0x7fa180004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.195+0000 7fa1957fa700 1 -- 172.21.15.67:0/1034919213 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa180003db0 con 0x7fa198109450 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.195+0000 7fa1957fa700 1 -- 172.21.15.67:0/1034919213 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa180011630 con 0x7fa198109450 2024-05-03T17:12:12.972 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.195+0000 7fa1957fa700 1 -- 172.21.15.67:0/1034919213 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa180011950 con 0x7fa198109450 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.195+0000 7fa19e38c700 1 -- 172.21.15.67:0/1034919213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1981a1820 con 0x7fa198109450 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.195+0000 7fa19e38c700 1 -- 172.21.15.67:0/1034919213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1981a1bc0 con 0x7fa198109450 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.195+0000 7fa1957fa700 1 -- 172.21.15.67:0/1034919213 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa18001a460 con 0x7fa198109450 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.195+0000 7fa1957fa700 1 -- 172.21.15.67:0/1034919213 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa18001b780 con 0x7fa198109450 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.196+0000 7fa19e38c700 1 -- 172.21.15.67:0/1034919213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa198064b00 con 0x7fa198109450 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.198+0000 7fa1957fa700 1 -- 172.21.15.67:0/1034919213 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa18001ba20 con 0x7fa198109450 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.271+0000 7fa19e38c700 1 -- 172.21.15.67:0/1034919213 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7fa19802e780 con 0x7fa198109450 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.271+0000 7fa1957fa700 1 -- 172.21.15.67:0/1034919213 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7fa18001bc00 con 0x7fa198109450 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.272+0000 7fa19e38c700 1 -- 172.21.15.67:0/1034919213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 msgr2=0x7fa19807d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.272+0000 7fa19e38c700 1 --2- 172.21.15.67:0/1034919213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 0x7fa19807d300 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fa180000c00 tx=0x7fa180004750 comp rx=0 tx=0).stop 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.272+0000 7fa19e38c700 1 -- 172.21.15.67:0/1034919213 shutdown_connections 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.272+0000 7fa19e38c700 1 --2- 172.21.15.67:0/1034919213 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa198109450 0x7fa19807d300 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.272+0000 7fa19e38c700 1 -- 172.21.15.67:0/1034919213 >> 172.21.15.67:0/1034919213 conn(0x7fa1981033f0 msgr2=0x7fa198192150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:12.973 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.272+0000 7fa19e38c700 1 -- 172.21.15.67:0/1034919213 shutdown_connections 2024-05-03T17:12:12.974 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:12.272+0000 7fa19e38c700 1 -- 172.21.15.67:0/1034919213 wait complete. 2024-05-03T17:12:12.974 INFO:teuthology.orchestra.run.smithi067.stdout:mon is available 2024-05-03T17:12:12.974 INFO:teuthology.orchestra.run.smithi067.stdout:Assimilating anything we can from ceph.conf... 2024-05-03T17:12:13.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:12 smithi067 ceph-mon[29359]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-03T17:12:13.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:12 smithi067 ceph-mon[29359]: monmap epoch 1 2024-05-03T17:12:13.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:12 smithi067 ceph-mon[29359]: fsid 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:12:13.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:12 smithi067 ceph-mon[29359]: last_changed 2024-05-03T17:12:08.010048+0000 2024-05-03T17:12:13.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:12 smithi067 ceph-mon[29359]: created 2024-05-03T17:12:08.010048+0000 2024-05-03T17:12:13.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:12 smithi067 ceph-mon[29359]: min_mon_release 19 (squid) 2024-05-03T17:12:13.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:12 smithi067 ceph-mon[29359]: election_strategy: 1 2024-05-03T17:12:13.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:12 smithi067 ceph-mon[29359]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-03T17:12:13.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:12 smithi067 ceph-mon[29359]: fsmap 2024-05-03T17:12:13.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:12 smithi067 ceph-mon[29359]: osdmap e1: 0 total, 0 up, 0 in 2024-05-03T17:12:13.078 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:12 smithi067 ceph-mon[29359]: mgrmap e1: no daemons active 2024-05-03T17:12:13.079 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:12 smithi067 ceph-mon[29359]: from='client.? 172.21.15.67:0/1034919213' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-03T17:12:13.974 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:13 smithi067 ceph-mon[29359]: from='client.? 172.21.15.67:0/3434307759' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-03T17:12:13.974 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:13 smithi067 ceph-mon[29359]: from='client.? 172.21.15.67:0/3434307759' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-03T17:12:13.984 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:13.984 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [global] 2024-05-03T17:12:13.984 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout fsid = 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:12:13.984 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-03T17:12:13.984 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.67:3300,v1:172.21.15.67:6789] 2024-05-03T17:12:13.984 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-03T17:12:13.984 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-03T17:12:13.984 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-03T17:12:13.984 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [osd] 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.343+0000 7f3051809700 1 Processor -- start 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.343+0000 7f3051809700 1 -- start start 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.343+0000 7f3051809700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 0x7f304c108220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:13.985 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.343+0000 7f3051809700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f304c108760 con 0x7f304c107e00 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.343+0000 7f304affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 0x7f304c108220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.343+0000 7f304affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 0x7f304c108220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35072/0 (socket says 172.21.15.67:35072) 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.343+0000 7f304affd700 1 -- 172.21.15.67:0/3151845875 learned_addr learned my addr 172.21.15.67:0/3151845875 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.343+0000 7f304affd700 1 -- 172.21.15.67:0/3151845875 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304c108f60 con 0x7f304c107e00 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.343+0000 7f304affd700 1 --2- 172.21.15.67:0/3151845875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 0x7f304c108220 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f30400098a0 tx=0x7f3040009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612e3d8b40dd971b server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.344+0000 7f3049ffb700 1 -- 172.21.15.67:0/3151845875 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3040004160 con 0x7f304c107e00 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.344+0000 7f3049ffb700 1 -- 172.21.15.67:0/3151845875 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3040004300 con 0x7f304c107e00 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.344+0000 7f3049ffb700 1 -- 172.21.15.67:0/3151845875 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30400036a0 con 0x7f304c107e00 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.344+0000 7f3051809700 1 -- 172.21.15.67:0/3151845875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 msgr2=0x7f304c108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.344+0000 7f3051809700 1 --2- 172.21.15.67:0/3151845875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 0x7f304c108220 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f30400098a0 tx=0x7f3040009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.344+0000 7f3051809700 1 -- 172.21.15.67:0/3151845875 shutdown_connections 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.344+0000 7f3051809700 1 --2- 172.21.15.67:0/3151845875 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 0x7f304c108220 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:13.986 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.344+0000 7f3051809700 1 -- 172.21.15.67:0/3151845875 >> 172.21.15.67:0/3151845875 conn(0x7f304c1033e0 msgr2=0x7f304c105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.344+0000 7f3051809700 1 -- 172.21.15.67:0/3151845875 shutdown_connections 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.344+0000 7f3051809700 1 -- 172.21.15.67:0/3151845875 wait complete. 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.345+0000 7f3051809700 1 Processor -- start 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.345+0000 7f3051809700 1 -- start start 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.345+0000 7f3051809700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 0x7f304c10dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.345+0000 7f3051809700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f304c109310 con 0x7f304c107e00 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.345+0000 7f304affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 0x7f304c10dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.346+0000 7f304affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 0x7f304c10dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35074/0 (socket says 172.21.15.67:35074) 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.346+0000 7f304affd700 1 -- 172.21.15.67:0/3434307759 learned_addr learned my addr 172.21.15.67:0/3434307759 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.346+0000 7f304affd700 1 -- 172.21.15.67:0/3434307759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304c10e1a0 con 0x7f304c107e00 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.346+0000 7f304affd700 1 --2- 172.21.15.67:0/3434307759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 0x7f304c10dc60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3040000c00 tx=0x7f3040004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.346+0000 7f3050807700 1 -- 172.21.15.67:0/3434307759 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3040003db0 con 0x7f304c107e00 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.346+0000 7f3050807700 1 -- 172.21.15.67:0/3434307759 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3040011630 con 0x7f304c107e00 2024-05-03T17:12:13.987 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.346+0000 7f3051809700 1 -- 172.21.15.67:0/3434307759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f304c10c3e0 con 0x7f304c107e00 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.346+0000 7f3050807700 1 -- 172.21.15.67:0/3434307759 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3040011950 con 0x7f304c107e00 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.346+0000 7f3051809700 1 -- 172.21.15.67:0/3434307759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f304c10c8c0 con 0x7f304c107e00 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.346+0000 7f3050807700 1 -- 172.21.15.67:0/3434307759 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f304001a460 con 0x7f304c107e00 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.346+0000 7f3050807700 1 -- 172.21.15.67:0/3434307759 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f304001b780 con 0x7f304c107e00 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.347+0000 7f3051809700 1 -- 172.21.15.67:0/3434307759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f304c064b00 con 0x7f304c107e00 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.349+0000 7f3050807700 1 -- 172.21.15.67:0/3434307759 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f304001ba20 con 0x7f304c107e00 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.420+0000 7f3051809700 1 -- 172.21.15.67:0/3434307759 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f304c196680 con 0x7f304c107e00 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.423+0000 7f3050807700 1 -- 172.21.15.67:0/3434307759 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+465 (secure 0 0 0) 0x7f304001bc00 con 0x7f304c107e00 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.424+0000 7f3050807700 1 -- 172.21.15.67:0/3434307759 <== mon.0 v2:172.21.15.67:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f304001bde0 con 0x7f304c107e00 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.424+0000 7f3051809700 1 -- 172.21.15.67:0/3434307759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 msgr2=0x7f304c10dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.424+0000 7f3051809700 1 --2- 172.21.15.67:0/3434307759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 0x7f304c10dc60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3040000c00 tx=0x7f3040004750 comp rx=0 tx=0).stop 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.425+0000 7f3051809700 1 -- 172.21.15.67:0/3434307759 shutdown_connections 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.425+0000 7f3051809700 1 --2- 172.21.15.67:0/3434307759 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f304c107e00 0x7f304c10dc60 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:13.988 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.425+0000 7f3051809700 1 -- 172.21.15.67:0/3434307759 >> 172.21.15.67:0/3434307759 conn(0x7f304c1033e0 msgr2=0x7f304c104050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:13.989 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.425+0000 7f3051809700 1 -- 172.21.15.67:0/3434307759 shutdown_connections 2024-05-03T17:12:13.989 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:13.425+0000 7f3051809700 1 -- 172.21.15.67:0/3434307759 wait complete. 2024-05-03T17:12:13.989 INFO:teuthology.orchestra.run.smithi067.stdout:Generating new minimal ceph.conf... 2024-05-03T17:12:15.014 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:14 smithi067 ceph-mon[29359]: from='client.? 172.21.15.67:0/774007884' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:12:15.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.413+0000 7f1034275700 1 Processor -- start 2024-05-03T17:12:15.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.413+0000 7f1034275700 1 -- start start 2024-05-03T17:12:15.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.414+0000 7f1034275700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 0x7f102c1075a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:15.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.414+0000 7f1034275700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f102c107b70 con 0x7f102c107180 2024-05-03T17:12:15.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.414+0000 7f1032011700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 0x7f102c1075a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:15.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.414+0000 7f1032011700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 0x7f102c1075a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35088/0 (socket says 172.21.15.67:35088) 2024-05-03T17:12:15.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.414+0000 7f1032011700 1 -- 172.21.15.67:0/3696102594 learned_addr learned my addr 172.21.15.67:0/3696102594 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:15.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.414+0000 7f1032011700 1 -- 172.21.15.67:0/3696102594 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102c10a160 con 0x7f102c107180 2024-05-03T17:12:15.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.414+0000 7f1032011700 1 --2- 172.21.15.67:0/3696102594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 0x7f102c1075a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f101c0098a0 tx=0x7f101c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b471d0ca66e2585 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:15.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.414+0000 7f103100f700 1 -- 172.21.15.67:0/3696102594 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f101c004160 con 0x7f102c107180 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.414+0000 7f103100f700 1 -- 172.21.15.67:0/3696102594 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f101c004300 con 0x7f102c107180 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.414+0000 7f103100f700 1 -- 172.21.15.67:0/3696102594 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f101c011af0 con 0x7f102c107180 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.415+0000 7f1034275700 1 -- 172.21.15.67:0/3696102594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 msgr2=0x7f102c1075a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.415+0000 7f1034275700 1 --2- 172.21.15.67:0/3696102594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 0x7f102c1075a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f101c0098a0 tx=0x7f101c009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.415+0000 7f1034275700 1 -- 172.21.15.67:0/3696102594 shutdown_connections 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.415+0000 7f1034275700 1 --2- 172.21.15.67:0/3696102594 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 0x7f102c1075a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.415+0000 7f1034275700 1 -- 172.21.15.67:0/3696102594 >> 172.21.15.67:0/3696102594 conn(0x7f102c102b00 msgr2=0x7f102c104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.415+0000 7f1034275700 1 -- 172.21.15.67:0/3696102594 shutdown_connections 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.415+0000 7f1034275700 1 -- 172.21.15.67:0/3696102594 wait complete. 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.416+0000 7f1034275700 1 Processor -- start 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.416+0000 7f1034275700 1 -- start start 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.416+0000 7f1034275700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 0x7f102c19cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.416+0000 7f1034275700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f102c10d390 con 0x7f102c107180 2024-05-03T17:12:15.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.416+0000 7f1032011700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 0x7f102c19cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.416+0000 7f1032011700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 0x7f102c19cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:35102/0 (socket says 172.21.15.67:35102) 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.416+0000 7f1032011700 1 -- 172.21.15.67:0/774007884 learned_addr learned my addr 172.21.15.67:0/774007884 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.416+0000 7f1032011700 1 -- 172.21.15.67:0/774007884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f102c19d320 con 0x7f102c107180 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.416+0000 7f1032011700 1 --2- 172.21.15.67:0/774007884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 0x7f102c19cde0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f101c003f40 tx=0x7f101c011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.416+0000 7f10237fe700 1 -- 172.21.15.67:0/774007884 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f101c0038b0 con 0x7f102c107180 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.417+0000 7f10237fe700 1 -- 172.21.15.67:0/774007884 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f101c003a50 con 0x7f102c107180 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.417+0000 7f1034275700 1 -- 172.21.15.67:0/774007884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f102c19d5b0 con 0x7f102c107180 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.417+0000 7f10237fe700 1 -- 172.21.15.67:0/774007884 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f101c01a5c0 con 0x7f102c107180 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.417+0000 7f1034275700 1 -- 172.21.15.67:0/774007884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f102c1a02a0 con 0x7f102c107180 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.417+0000 7f10237fe700 1 -- 172.21.15.67:0/774007884 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f101c01ac00 con 0x7f102c107180 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.417+0000 7f10237fe700 1 -- 172.21.15.67:0/774007884 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f101c019700 con 0x7f102c107180 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.418+0000 7f1034275700 1 -- 172.21.15.67:0/774007884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f102c064b00 con 0x7f102c107180 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.420+0000 7f10237fe700 1 -- 172.21.15.67:0/774007884 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f101c019900 con 0x7f102c107180 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.491+0000 7f1034275700 1 -- 172.21.15.67:0/774007884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f102c1a0470 con 0x7f102c107180 2024-05-03T17:12:15.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.491+0000 7f10237fe700 1 -- 172.21.15.67:0/774007884 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f101c019ae0 con 0x7f102c107180 2024-05-03T17:12:15.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.492+0000 7f1034275700 1 -- 172.21.15.67:0/774007884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 msgr2=0x7f102c19cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:15.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.492+0000 7f1034275700 1 --2- 172.21.15.67:0/774007884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 0x7f102c19cde0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f101c003f40 tx=0x7f101c011890 comp rx=0 tx=0).stop 2024-05-03T17:12:15.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.492+0000 7f1034275700 1 -- 172.21.15.67:0/774007884 shutdown_connections 2024-05-03T17:12:15.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.492+0000 7f1034275700 1 --2- 172.21.15.67:0/774007884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f102c107180 0x7f102c19cde0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:15.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.492+0000 7f1034275700 1 -- 172.21.15.67:0/774007884 >> 172.21.15.67:0/774007884 conn(0x7f102c102b00 msgr2=0x7f102c103640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:15.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.492+0000 7f1034275700 1 -- 172.21.15.67:0/774007884 shutdown_connections 2024-05-03T17:12:15.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:14.492+0000 7f1034275700 1 -- 172.21.15.67:0/774007884 wait complete. 2024-05-03T17:12:15.025 INFO:teuthology.orchestra.run.smithi067.stdout:Restarting the monitor... 2024-05-03T17:12:15.293 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:15 smithi067 systemd[1]: Stopping Ceph mon.a for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:12:15.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:15 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a[29355]: 2024-05-03T17:12:15.291+0000 7fed3c1ae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:12:15.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:15 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a[29355]: 2024-05-03T17:12:15.291+0000 7fed3c1ae700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-03T17:12:15.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:15 smithi067 podman[29603]: 2024-05-03 17:12:15.4862662 +0000 UTC m=+0.301693810 container died 46de02592d24395619dd6da357646c3f3cbef71d7fd16306a1229f1817be569e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-03T17:12:16.109 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:15 smithi067 podman[29603]: 2024-05-03 17:12:15.841716198 +0000 UTC m=+0.657143807 container cleanup 46de02592d24395619dd6da357646c3f3cbef71d7fd16306a1229f1817be569e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-03T17:12:16.109 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:15 smithi067 bash[29603]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a 2024-05-03T17:12:16.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 podman[29617]: 2024-05-03 17:12:16.109091025 +0000 UTC m=+0.618793394 container remove 46de02592d24395619dd6da357646c3f3cbef71d7fd16306a1229f1817be569e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:12:16.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 systemd[1]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-03T17:12:16.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 systemd[1]: Stopped Ceph mon.a for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:12:16.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 systemd[1]: Starting Ceph mon.a for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:12:16.838 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 podman[29712]: 2024-05-03 17:12:16.393756188 +0000 UTC m=+0.018227805 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T17:12:16.838 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 podman[29712]: 2024-05-03 17:12:16.493418871 +0000 UTC m=+0.117890502 container create 60217e9ce4fcdb6735b0435e89a0320b40ce4733d201df85abad1c2f6054fd72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2) 2024-05-03T17:12:16.838 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 podman[29712]: 2024-05-03 17:12:16.837492056 +0000 UTC m=+0.461963652 container init 60217e9ce4fcdb6735b0435e89a0320b40ce4733d201df85abad1c2f6054fd72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:12:16.925 INFO:teuthology.orchestra.run.smithi067.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-03T17:12:17.166 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 podman[29712]: 2024-05-03 17:12:16.840779871 +0000 UTC m=+0.465251457 container start 60217e9ce4fcdb6735b0435e89a0320b40ce4733d201df85abad1c2f6054fd72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430) 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: set uid:gid to 167:167 (ceph:ceph) 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: ceph version 19.0.0-3450-g9a047fe7 (9a047fe7599a549042d06a3fb5b6813a203eabea) squid (dev), process ceph-mon, pid 2 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: pidfile_write: ignore empty --pid-file 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: load: jerasure load: lrc 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: RocksDB version: 7.9.2 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Git sha 0 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Compile date 2024-05-01 21:12:03 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: DB SUMMARY 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: DB Session ID: IB64T8VPYNZ754RXXWY5 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: CURRENT file: CURRENT 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: IDENTITY file: IDENTITY 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-03T17:12:17.167 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105208 ; 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.error_if_exists: 0 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.create_if_missing: 0 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.paranoid_checks: 1 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.env: 0x55bd54ca6cc0 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.fs: PosixFileSystem 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.info_log: 0x55bd55c07da0 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_file_opening_threads: 16 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.statistics: (nil) 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.use_fsync: 0 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_log_file_size: 0 2024-05-03T17:12:17.168 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.keep_log_file_num: 1000 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.recycle_log_file_num: 0 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.allow_fallocate: 1 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.allow_mmap_reads: 0 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.allow_mmap_writes: 0 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.use_direct_reads: 0 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.create_missing_column_families: 0 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.db_log_dir: 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.wal_dir: 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-03T17:12:17.169 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.advise_random_on_open: 1 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.db_write_buffer_size: 0 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.write_buffer_manager: 0x55bd55bd90e0 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.rate_limiter: (nil) 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.wal_recovery_mode: 2 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.enable_thread_tracking: 0 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.enable_pipelined_write: 0 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.unordered_write: 0 2024-05-03T17:12:17.170 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.row_cache: None 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.wal_filter: None 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.allow_ingest_behind: 0 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.two_write_queues: 0 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.manual_wal_flush: 0 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.wal_compression: 0 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.atomic_flush: 0 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-03T17:12:17.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-03T17:12:17.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.log_readahead_size: 0 2024-05-03T17:12:17.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-03T17:12:17.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.best_efforts_recovery: 0 2024-05-03T17:12:17.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-03T17:12:17.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-03T17:12:17.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.allow_data_in_errors: 0 2024-05-03T17:12:17.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.db_host_id: __hostname__ 2024-05-03T17:12:17.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-03T17:12:17.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_background_jobs: 2 2024-05-03T17:12:17.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_background_compactions: -1 2024-05-03T17:12:17.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_subcompactions: 1 2024-05-03T17:12:17.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_total_wal_size: 0 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_open_files: -1 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bytes_per_sync: 0 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_readahead_size: 0 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_background_flushes: -1 2024-05-03T17:12:17.174 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Compression algorithms supported: 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: kZSTD supported: 0 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: kXpressCompression supported: 0 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: kLZ4HCCompression supported: 1 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: kZlibCompression supported: 1 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: kSnappyCompression supported: 1 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: kLZ4Compression supported: 1 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: kBZip2Compression supported: 0 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.merge_operator: 2024-05-03T17:12:17.175 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_filter: None 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_filter_factory: None 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.sst_partitioner_factory: None 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55bd55c07f60) 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout: cache_index_and_filter_blocks: 1 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout: pin_top_level_index_and_filter: 1 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_type: 0 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout: data_block_index_type: 0 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_shortening: 1 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout: checksum: 4 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout: no_block_cache: 0 2024-05-03T17:12:17.176 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache: 0x55bd55c0b1f0 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_name: BinnedLRUCache 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_options: 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: capacity : 536870912 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: num_shard_bits : 4 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: strict_capacity_limit : 0 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: high_pri_pool_ratio: 0.000 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_cache_compressed: (nil) 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: persistent_cache: (nil) 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_size: 4096 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_size_deviation: 10 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_restart_interval: 16 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: index_block_restart_interval: 1 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: metadata_block_size: 4096 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: partition_filters: 0 2024-05-03T17:12:17.177 INFO:journalctl@ceph.mon.a.smithi067.stdout: use_delta_encoding: 1 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout: filter_policy: bloomfilter 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout: whole_key_filtering: 1 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout: verify_compression: 0 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout: read_amp_bytes_per_bit: 0 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout: format_version: 5 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout: enable_index_compression: 1 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout: block_align: 0 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout: max_auto_readahead_size: 262144 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout: prepopulate_block_cache: 0 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout: initial_auto_readahead_size: 8192 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout: num_file_reads_for_auto_readahead: 2 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.write_buffer_size: 33554432 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_write_buffer_number: 2 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compression: NoCompression 2024-05-03T17:12:17.178 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bottommost_compression: Disabled 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.prefix_extractor: nullptr 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.num_levels: 7 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-03T17:12:17.179 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compression_opts.level: 32767 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compression_opts.strategy: 0 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compression_opts.enabled: false 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.target_file_size_base: 67108864 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-03T17:12:17.180 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.arena_block_size: 1048576 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-03T17:12:17.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.disable_auto_compactions: 0 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.inplace_update_support: 0 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-03T17:12:17.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.bloom_locality: 0 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.max_successive_merges: 0 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.paranoid_file_checks: 0 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.force_consistency_checks: 1 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.report_bg_io_stats: 0 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.ttl: 2592000 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.enable_blob_files: false 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.min_blob_size: 0 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.blob_file_size: 268435456 2024-05-03T17:12:17.183 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.blob_file_starting_level: 0 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 78461725-711e-48b6-b732-53d667aca472 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714756336872449, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714756336873645, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100359, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 98253, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14191, "raw_average_key_size": 49, "raw_value_size": 90522, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714756336, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "78461725-711e-48b6-b732-53d667aca472", "db_session_id": "IB64T8VPYNZ754RXXWY5", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714756336873744, "job": 1, "event": "recovery_finished"} 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-03T17:12:17.184 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55bd55d28000 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: DB pointer 0x55bd55d14000 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** DB Stats ** 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-03T17:12:17.185 INFO:journalctl@ceph.mon.a.smithi067.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: L0 2/0 99.87 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: Sum 2/0 99.87 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 125.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** Compaction Stats [default] ** 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 125.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-03T17:12:17.186 INFO:journalctl@ceph.mon.a.smithi067.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout: Cumulative compaction: 0.00 GB write, 28.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout: Interval compaction: 0.00 GB write, 28.90 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout: Block cache BinnedLRUCache@0x55bd55c0b1f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: starting mon.a rank 0 at public addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] at bind addrs [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: mon.a@-1(???) e1 preinit fsid 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: mon.a@-1(???).mds e1 new map 2024-05-03T17:12:17.187 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: mon.a@-1(???).mds e1 print_map 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout: e1 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout: legacy client fscid: -1 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout: 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout: No filesystems configured 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 bash[29712]: 60217e9ce4fcdb6735b0435e89a0320b40ce4733d201df85abad1c2f6054fd72 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 systemd[1]: Started Ceph mon.a for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: monmap epoch 1 2024-05-03T17:12:17.188 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: fsid 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:12:17.189 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: last_changed 2024-05-03T17:12:08.010048+0000 2024-05-03T17:12:17.189 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: created 2024-05-03T17:12:08.010048+0000 2024-05-03T17:12:17.189 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: min_mon_release 19 (squid) 2024-05-03T17:12:17.189 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: election_strategy: 1 2024-05-03T17:12:17.189 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-03T17:12:17.189 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: fsmap 2024-05-03T17:12:17.189 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: osdmap e1: 0 total, 0 up, 0 in 2024-05-03T17:12:17.189 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:16 smithi067 ceph-mon[29731]: mgrmap e1: no daemons active 2024-05-03T17:12:17.934 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.388+0000 7f98fe742700 1 Processor -- start 2024-05-03T17:12:17.934 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.388+0000 7f98fe742700 1 -- start start 2024-05-03T17:12:17.934 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.388+0000 7f98fe742700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 0x7f98f81081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:17.934 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.388+0000 7f98fe742700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f98f81087b0 con 0x7f98f8107dc0 2024-05-03T17:12:17.934 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.389+0000 7f98f7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 0x7f98f81081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:17.934 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.389+0000 7f98f7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 0x7f98f81081e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33432/0 (socket says 172.21.15.67:33432) 2024-05-03T17:12:17.934 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.389+0000 7f98f7fff700 1 -- 172.21.15.67:0/3266489602 learned_addr learned my addr 172.21.15.67:0/3266489602 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:17.934 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.389+0000 7f98f7fff700 1 -- 172.21.15.67:0/3266489602 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f8108fc0 con 0x7f98f8107dc0 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.389+0000 7f98f7fff700 1 --2- 172.21.15.67:0/3266489602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 0x7f98f81081e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f98e80098a0 tx=0x7f98e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754f667c8a671610 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.389+0000 7f98f6ffd700 1 -- 172.21.15.67:0/3266489602 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e8004160 con 0x7f98f8107dc0 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.390+0000 7f98f6ffd700 1 -- 172.21.15.67:0/3266489602 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f98e8004300 con 0x7f98f8107dc0 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.390+0000 7f98f6ffd700 1 -- 172.21.15.67:0/3266489602 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e8011af0 con 0x7f98f8107dc0 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.390+0000 7f98fe742700 1 -- 172.21.15.67:0/3266489602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 msgr2=0x7f98f81081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.390+0000 7f98fe742700 1 --2- 172.21.15.67:0/3266489602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 0x7f98f81081e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f98e80098a0 tx=0x7f98e8009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.390+0000 7f98fe742700 1 -- 172.21.15.67:0/3266489602 shutdown_connections 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.390+0000 7f98fe742700 1 --2- 172.21.15.67:0/3266489602 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 0x7f98f81081e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.390+0000 7f98fe742700 1 -- 172.21.15.67:0/3266489602 >> 172.21.15.67:0/3266489602 conn(0x7f98f8103380 msgr2=0x7f98f81057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.390+0000 7f98fe742700 1 -- 172.21.15.67:0/3266489602 shutdown_connections 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.390+0000 7f98fe742700 1 -- 172.21.15.67:0/3266489602 wait complete. 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.391+0000 7f98fe742700 1 Processor -- start 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.391+0000 7f98fe742700 1 -- start start 2024-05-03T17:12:17.935 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.391+0000 7f98fe742700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 0x7f98f8192eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.391+0000 7f98fe742700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f98f810c1f0 con 0x7f98f8107dc0 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.391+0000 7f98f7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 0x7f98f8192eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.391+0000 7f98f7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 0x7f98f8192eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33434/0 (socket says 172.21.15.67:33434) 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.391+0000 7f98f7fff700 1 -- 172.21.15.67:0/1492102640 learned_addr learned my addr 172.21.15.67:0/1492102640 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.392+0000 7f98f7fff700 1 -- 172.21.15.67:0/1492102640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f81933f0 con 0x7f98f8107dc0 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.392+0000 7f98f7fff700 1 --2- 172.21.15.67:0/1492102640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 0x7f98f8192eb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f98e8000c00 tx=0x7f98e8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.392+0000 7f98f57fa700 1 -- 172.21.15.67:0/1492102640 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e8003b20 con 0x7f98f8107dc0 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.392+0000 7f98f57fa700 1 -- 172.21.15.67:0/1492102640 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f98e801a8c0 con 0x7f98f8107dc0 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.392+0000 7f98fe742700 1 -- 172.21.15.67:0/1492102640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98f81915e0 con 0x7f98f8107dc0 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.392+0000 7f98f57fa700 1 -- 172.21.15.67:0/1492102640 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98e8024940 con 0x7f98f8107dc0 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.392+0000 7f98fe742700 1 -- 172.21.15.67:0/1492102640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98f8191b20 con 0x7f98f8107dc0 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.392+0000 7f98f57fa700 1 -- 172.21.15.67:0/1492102640 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f98e8022020 con 0x7f98f8107dc0 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.392+0000 7f98f57fa700 1 -- 172.21.15.67:0/1492102640 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f98e802ddd0 con 0x7f98f8107dc0 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.393+0000 7f98fe742700 1 -- 172.21.15.67:0/1492102640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98f810c370 con 0x7f98f8107dc0 2024-05-03T17:12:17.936 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.395+0000 7f98f57fa700 1 -- 172.21.15.67:0/1492102640 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f98e8042e50 con 0x7f98f8107dc0 2024-05-03T17:12:17.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.465+0000 7f98fe742700 1 -- 172.21.15.67:0/1492102640 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f98f8064b00 con 0x7f98f8107dc0 2024-05-03T17:12:17.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.467+0000 7f98f57fa700 1 -- 172.21.15.67:0/1492102640 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f98e8031020 con 0x7f98f8107dc0 2024-05-03T17:12:17.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.468+0000 7f98fe742700 1 -- 172.21.15.67:0/1492102640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 msgr2=0x7f98f8192eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:17.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.468+0000 7f98fe742700 1 --2- 172.21.15.67:0/1492102640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 0x7f98f8192eb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f98e8000c00 tx=0x7f98e8011b70 comp rx=0 tx=0).stop 2024-05-03T17:12:17.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.468+0000 7f98fe742700 1 -- 172.21.15.67:0/1492102640 shutdown_connections 2024-05-03T17:12:17.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.468+0000 7f98fe742700 1 --2- 172.21.15.67:0/1492102640 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98f8107dc0 0x7f98f8192eb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:17.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.468+0000 7f98fe742700 1 -- 172.21.15.67:0/1492102640 >> 172.21.15.67:0/1492102640 conn(0x7f98f8103380 msgr2=0x7f98f8104250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:17.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.468+0000 7f98fe742700 1 -- 172.21.15.67:0/1492102640 shutdown_connections 2024-05-03T17:12:17.937 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:17.468+0000 7f98fe742700 1 -- 172.21.15.67:0/1492102640 wait complete. 2024-05-03T17:12:17.937 INFO:teuthology.orchestra.run.smithi067.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-03T17:12:17.960 INFO:teuthology.orchestra.run.smithi067.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-03T17:12:17.960 INFO:teuthology.orchestra.run.smithi067.stdout:Creating mgr... 2024-05-03T17:12:17.960 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-03T17:12:17.961 INFO:teuthology.orchestra.run.smithi067.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-03T17:12:18.199 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.x 2024-05-03T17:12:18.200 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Failed to reset failed state of unit ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.x.service: Unit ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.x.service not loaded. 2024-05-03T17:12:18.356 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-20c8e22e-0970-11ef-bc97-c7b262605968.target.wants/ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-20c8e22e-0970-11ef-bc97-c7b262605968@.service. 2024-05-03T17:12:18.468 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:18 smithi067 systemd[1]: Starting Ceph mgr.x for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:12:18.719 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:18 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1492102640' entity='client.admin' 2024-05-03T17:12:18.719 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:18 smithi067 podman[29947]: 2024-05-03 17:12:18.497300736 +0000 UTC m=+0.018815942 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea 2024-05-03T17:12:18.719 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:18 smithi067 podman[29947]: 2024-05-03 17:12:18.595695282 +0000 UTC m=+0.117210474 container create 0a733d8f609c66daba9bbd65dfafaac05d7969582a420d487ff5c1d8e7a77c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-05-03T17:12:18.895 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-03T17:12:18.895 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-05-03T17:12:18.902 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-03T17:12:18.902 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-05-03T17:12:18.902 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-05-03T17:12:18.902 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to enable service . firewalld.service is not available 2024-05-03T17:12:18.911 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-03T17:12:18.911 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-05-03T17:12:18.918 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-03T17:12:18.918 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-05-03T17:12:18.918 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-05-03T17:12:18.918 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-03T17:12:18.918 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr to start... 2024-05-03T17:12:18.918 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr... 2024-05-03T17:12:19.082 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:18 smithi067 podman[29947]: 2024-05-03 17:12:18.806340625 +0000 UTC m=+0.327855817 container init 0a733d8f609c66daba9bbd65dfafaac05d7969582a420d487ff5c1d8e7a77c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:12:19.083 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:18 smithi067 podman[29947]: 2024-05-03 17:12:18.808904878 +0000 UTC m=+0.330420070 container start 0a733d8f609c66daba9bbd65dfafaac05d7969582a420d487ff5c1d8e7a77c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True) 2024-05-03T17:12:19.083 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:18 smithi067 bash[29947]: 0a733d8f609c66daba9bbd65dfafaac05d7969582a420d487ff5c1d8e7a77c97 2024-05-03T17:12:19.083 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:18 smithi067 systemd[1]: Started Ceph mgr.x for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:12:19.083 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:18 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:18.978+0000 7f3341d12200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T17:12:19.384 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:19 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:19.171+0000 7f3341d12200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T17:12:19.385 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:19 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:19.281+0000 7f3341d12200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T17:12:19.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:19 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3238526025' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-03T17:12:19.703 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:19 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:19.631+0000 7f3341d12200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T17:12:20.005 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:19 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:19.721+0000 7f3341d12200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T17:12:20.007 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:20.018 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T17:12:20.018 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "20c8e22e-0970-11ef-bc97-c7b262605968", 2024-05-03T17:12:20.018 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-05-03T17:12:20.018 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-03T17:12:20.018 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-03T17:12:20.018 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-03T17:12:20.018 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:20.018 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "a" 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-03T17:12:20.019 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-03T17:12:20.020 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T17:12:20.021 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-05-03T17:12:11.709878+0000", 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.395+0000 7fdf80833700 1 Processor -- start 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.395+0000 7fdf80833700 1 -- start start 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.396+0000 7fdf80833700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 0x7fdf7c1092f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.396+0000 7fdf80833700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c109830 con 0x7fdf7c108ed0 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.396+0000 7fdf7a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 0x7fdf7c1092f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.396+0000 7fdf7a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 0x7fdf7c1092f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33456/0 (socket says 172.21.15.67:33456) 2024-05-03T17:12:20.022 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.396+0000 7fdf7a59c700 1 -- 172.21.15.67:0/2724168878 learned_addr learned my addr 172.21.15.67:0/2724168878 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.396+0000 7fdf7a59c700 1 -- 172.21.15.67:0/2724168878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c10a070 con 0x7fdf7c108ed0 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.397+0000 7fdf7a59c700 1 --2- 172.21.15.67:0/2724168878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 0x7fdf7c1092f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0098a0 tx=0x7fdf6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d355b9c6f4dbf1ca server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.397+0000 7fdf7959a700 1 -- 172.21.15.67:0/2724168878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf6c004160 con 0x7fdf7c108ed0 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.397+0000 7fdf7959a700 1 -- 172.21.15.67:0/2724168878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdf6c004300 con 0x7fdf7c108ed0 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.397+0000 7fdf7959a700 1 -- 172.21.15.67:0/2724168878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf6c011af0 con 0x7fdf7c108ed0 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.397+0000 7fdf80833700 1 -- 172.21.15.67:0/2724168878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 msgr2=0x7fdf7c1092f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.398+0000 7fdf80833700 1 --2- 172.21.15.67:0/2724168878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 0x7fdf7c1092f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c0098a0 tx=0x7fdf6c009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.398+0000 7fdf80833700 1 -- 172.21.15.67:0/2724168878 shutdown_connections 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.398+0000 7fdf80833700 1 --2- 172.21.15.67:0/2724168878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 0x7fdf7c1092f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.398+0000 7fdf80833700 1 -- 172.21.15.67:0/2724168878 >> 172.21.15.67:0/2724168878 conn(0x7fdf7c1044e0 msgr2=0x7fdf7c106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.398+0000 7fdf80833700 1 -- 172.21.15.67:0/2724168878 shutdown_connections 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.398+0000 7fdf80833700 1 -- 172.21.15.67:0/2724168878 wait complete. 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.399+0000 7fdf80833700 1 Processor -- start 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.399+0000 7fdf80833700 1 -- start start 2024-05-03T17:12:20.023 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.399+0000 7fdf80833700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 0x7fdf7c19ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.399+0000 7fdf80833700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c1099b0 con 0x7fdf7c108ed0 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.399+0000 7fdf7a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 0x7fdf7c19ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.399+0000 7fdf7a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 0x7fdf7c19ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33460/0 (socket says 172.21.15.67:33460) 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.399+0000 7fdf7a59c700 1 -- 172.21.15.67:0/3238526025 learned_addr learned my addr 172.21.15.67:0/3238526025 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.399+0000 7fdf7a59c700 1 -- 172.21.15.67:0/3238526025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c19d390 con 0x7fdf7c108ed0 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.400+0000 7fdf7a59c700 1 --2- 172.21.15.67:0/3238526025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 0x7fdf7c19ce50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009fb0 tx=0x7fdf6c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.400+0000 7fdf6b7fe700 1 -- 172.21.15.67:0/3238526025 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf6c0036a0 con 0x7fdf7c108ed0 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.400+0000 7fdf6b7fe700 1 -- 172.21.15.67:0/3238526025 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fdf6c003cd0 con 0x7fdf7c108ed0 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.400+0000 7fdf80833700 1 -- 172.21.15.67:0/3238526025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7c19d620 con 0x7fdf7c108ed0 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.400+0000 7fdf6b7fe700 1 -- 172.21.15.67:0/3238526025 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdf6c0193c0 con 0x7fdf7c108ed0 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.400+0000 7fdf80833700 1 -- 172.21.15.67:0/3238526025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf7c1a0310 con 0x7fdf7c108ed0 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.400+0000 7fdf6b7fe700 1 -- 172.21.15.67:0/3238526025 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdf6c019a00 con 0x7fdf7c108ed0 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.400+0000 7fdf6b7fe700 1 -- 172.21.15.67:0/3238526025 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fdf6c024730 con 0x7fdf7c108ed0 2024-05-03T17:12:20.024 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.401+0000 7fdf80833700 1 -- 172.21.15.67:0/3238526025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf7c064b00 con 0x7fdf7c108ed0 2024-05-03T17:12:20.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.403+0000 7fdf6b7fe700 1 -- 172.21.15.67:0/3238526025 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fdf6c01a430 con 0x7fdf7c108ed0 2024-05-03T17:12:20.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.492+0000 7fdf80833700 1 -- 172.21.15.67:0/3238526025 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fdf7c1a04e0 con 0x7fdf7c108ed0 2024-05-03T17:12:20.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.492+0000 7fdf6b7fe700 1 -- 172.21.15.67:0/3238526025 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fdf6c032030 con 0x7fdf7c108ed0 2024-05-03T17:12:20.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.493+0000 7fdf80833700 1 -- 172.21.15.67:0/3238526025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 msgr2=0x7fdf7c19ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:20.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.493+0000 7fdf80833700 1 --2- 172.21.15.67:0/3238526025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 0x7fdf7c19ce50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fdf6c009fb0 tx=0x7fdf6c011940 comp rx=0 tx=0).stop 2024-05-03T17:12:20.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.493+0000 7fdf80833700 1 -- 172.21.15.67:0/3238526025 shutdown_connections 2024-05-03T17:12:20.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.493+0000 7fdf80833700 1 --2- 172.21.15.67:0/3238526025 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fdf7c108ed0 0x7fdf7c19ce50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:20.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.493+0000 7fdf80833700 1 -- 172.21.15.67:0/3238526025 >> 172.21.15.67:0/3238526025 conn(0x7fdf7c1044e0 msgr2=0x7fdf7c1050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:20.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.494+0000 7fdf80833700 1 -- 172.21.15.67:0/3238526025 shutdown_connections 2024-05-03T17:12:20.025 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:19.494+0000 7fdf80833700 1 -- 172.21.15.67:0/3238526025 wait complete. 2024-05-03T17:12:20.025 INFO:teuthology.orchestra.run.smithi067.stdout:mgr not available, waiting (1/15)... 2024-05-03T17:12:21.342 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:20 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:20.989+0000 7f3341d12200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T17:12:21.342 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:21 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:21.164+0000 7f3341d12200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T17:12:21.342 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:21 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:21.341+0000 7f3341d12200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T17:12:21.945 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:21 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:21.640+0000 7f3341d12200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T17:12:21.945 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:21 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:21.735+0000 7f3341d12200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T17:12:21.945 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:21 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:21.944+0000 7f3341d12200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T17:12:22.204 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:22 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:22.148+0000 7f3341d12200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T17:12:22.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:22 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2222755461' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-03T17:12:22.953 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:22 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:22.801+0000 7f3341d12200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T17:12:23.074 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:23.074 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T17:12:23.074 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "20c8e22e-0970-11ef-bc97-c7b262605968", 2024-05-03T17:12:23.074 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-05-03T17:12:23.074 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-03T17:12:23.074 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-03T17:12:23.074 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-03T17:12:23.074 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:23.074 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-03T17:12:23.074 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "a" 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:23.075 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-03T17:12:23.076 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T17:12:23.077 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-05-03T17:12:11.709878+0000", 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.477+0000 7f7419dde700 1 Processor -- start 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.477+0000 7f7419dde700 1 -- start start 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.477+0000 7f7419dde700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 0x7f741410a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.477+0000 7f7419dde700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f741410aea0 con 0x7f741410a540 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.477+0000 7f74137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 0x7f741410a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:23.078 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.477+0000 7f74137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 0x7f741410a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33470/0 (socket says 172.21.15.67:33470) 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.477+0000 7f74137fe700 1 -- 172.21.15.67:0/564440857 learned_addr learned my addr 172.21.15.67:0/564440857 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.477+0000 7f74137fe700 1 -- 172.21.15.67:0/564440857 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7414108bc0 con 0x7f741410a540 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.478+0000 7f74137fe700 1 --2- 172.21.15.67:0/564440857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 0x7f741410a960 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f74040098a0 tx=0x7f7404009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26703a248a3226c7 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.478+0000 7f74127fc700 1 -- 172.21.15.67:0/564440857 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7404004160 con 0x7f741410a540 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.478+0000 7f74127fc700 1 -- 172.21.15.67:0/564440857 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7404004300 con 0x7f741410a540 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.478+0000 7f74127fc700 1 -- 172.21.15.67:0/564440857 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7404011af0 con 0x7f741410a540 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.478+0000 7f7419dde700 1 -- 172.21.15.67:0/564440857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 msgr2=0x7f741410a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.478+0000 7f7419dde700 1 --2- 172.21.15.67:0/564440857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 0x7f741410a960 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f74040098a0 tx=0x7f7404009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.479+0000 7f7419dde700 1 -- 172.21.15.67:0/564440857 shutdown_connections 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.479+0000 7f7419dde700 1 --2- 172.21.15.67:0/564440857 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 0x7f741410a960 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.479+0000 7f7419dde700 1 -- 172.21.15.67:0/564440857 >> 172.21.15.67:0/564440857 conn(0x7f7414104580 msgr2=0x7f74141069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.479+0000 7f7419dde700 1 -- 172.21.15.67:0/564440857 shutdown_connections 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.479+0000 7f7419dde700 1 -- 172.21.15.67:0/564440857 wait complete. 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.479+0000 7f7419dde700 1 Processor -- start 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.479+0000 7f7419dde700 1 -- start start 2024-05-03T17:12:23.079 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.480+0000 7f7419dde700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 0x7f741419cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.480+0000 7f7419dde700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f741410d320 con 0x7f741410a540 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.480+0000 7f74137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 0x7f741419cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.480+0000 7f74137fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 0x7f741419cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33486/0 (socket says 172.21.15.67:33486) 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.480+0000 7f74137fe700 1 -- 172.21.15.67:0/2222755461 learned_addr learned my addr 172.21.15.67:0/2222755461 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.480+0000 7f74137fe700 1 -- 172.21.15.67:0/2222755461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741419d430 con 0x7f741410a540 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.480+0000 7f74137fe700 1 --2- 172.21.15.67:0/2222755461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 0x7f741419cef0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7404009fb0 tx=0x7f7404011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.480+0000 7f7410ff9700 1 -- 172.21.15.67:0/2222755461 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74040036a0 con 0x7f741410a540 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.480+0000 7f7419dde700 1 -- 172.21.15.67:0/2222755461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f741419d6c0 con 0x7f741410a540 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.480+0000 7f7410ff9700 1 -- 172.21.15.67:0/2222755461 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f740401a430 con 0x7f741410a540 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.480+0000 7f7410ff9700 1 -- 172.21.15.67:0/2222755461 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7404019550 con 0x7f741410a540 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.481+0000 7f7419dde700 1 -- 172.21.15.67:0/2222755461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74141a03b0 con 0x7f741410a540 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.481+0000 7f7410ff9700 1 -- 172.21.15.67:0/2222755461 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7404019be0 con 0x7f741410a540 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.481+0000 7f7410ff9700 1 -- 172.21.15.67:0/2222755461 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7404024730 con 0x7f741410a540 2024-05-03T17:12:23.080 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.482+0000 7f7419dde700 1 -- 172.21.15.67:0/2222755461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7414064b00 con 0x7f741410a540 2024-05-03T17:12:23.081 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.484+0000 7f7410ff9700 1 -- 172.21.15.67:0/2222755461 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f7404043e50 con 0x7f741410a540 2024-05-03T17:12:23.081 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.559+0000 7f7419dde700 1 -- 172.21.15.67:0/2222755461 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f74141a0810 con 0x7f741410a540 2024-05-03T17:12:23.081 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.559+0000 7f7410ff9700 1 -- 172.21.15.67:0/2222755461 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f7404032030 con 0x7f741410a540 2024-05-03T17:12:23.081 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.560+0000 7f7419dde700 1 -- 172.21.15.67:0/2222755461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 msgr2=0x7f741419cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:23.081 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.560+0000 7f7419dde700 1 --2- 172.21.15.67:0/2222755461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 0x7f741419cef0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7404009fb0 tx=0x7f7404011940 comp rx=0 tx=0).stop 2024-05-03T17:12:23.081 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.560+0000 7f7419dde700 1 -- 172.21.15.67:0/2222755461 shutdown_connections 2024-05-03T17:12:23.081 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.560+0000 7f7419dde700 1 --2- 172.21.15.67:0/2222755461 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f741410a540 0x7f741419cef0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:23.081 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.560+0000 7f7419dde700 1 -- 172.21.15.67:0/2222755461 >> 172.21.15.67:0/2222755461 conn(0x7f7414104580 msgr2=0x7f7414105110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:23.081 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.561+0000 7f7419dde700 1 -- 172.21.15.67:0/2222755461 shutdown_connections 2024-05-03T17:12:23.081 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:22.561+0000 7f7419dde700 1 -- 172.21.15.67:0/2222755461 wait complete. 2024-05-03T17:12:23.081 INFO:teuthology.orchestra.run.smithi067.stdout:mgr not available, waiting (2/15)... 2024-05-03T17:12:23.453 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:23 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:23.030+0000 7f3341d12200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-03T17:12:23.953 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:23 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:23.469+0000 7f3341d12200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T17:12:23.953 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:23 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:23.569+0000 7f3341d12200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T17:12:24.343 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:23 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:23.972+0000 7f3341d12200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-03T17:12:24.344 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:24 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:24.343+0000 7f3341d12200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T17:12:24.690 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:24 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:24.438+0000 7f3341d12200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T17:12:24.953 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:24 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:24.689+0000 7f3341d12200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T17:12:24.954 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:24 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:24.827+0000 7f3341d12200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T17:12:25.453 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:25 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:25.144+0000 7f3341d12200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T17:12:25.453 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:25 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:25.264+0000 7f3341d12200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T17:12:25.453 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:25 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:25.363+0000 7f3341d12200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T17:12:25.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:25 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1279907978' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-03T17:12:26.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:26.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T17:12:26.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "20c8e22e-0970-11ef-bc97-c7b262605968", 2024-05-03T17:12:26.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "a" 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:26.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-03T17:12:26.157 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-03T17:12:26.158 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-05-03T17:12:11.709878+0000", 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.489+0000 7f1399369700 1 Processor -- start 2024-05-03T17:12:26.159 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.489+0000 7f1399369700 1 -- start start 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.489+0000 7f1399369700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 0x7f1394108210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.489+0000 7f1399369700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f13941087e0 con 0x7f1394107df0 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.489+0000 7f1392ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 0x7f1394108210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.490+0000 7f1392ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 0x7f1394108210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33494/0 (socket says 172.21.15.67:33494) 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.490+0000 7f1392ffd700 1 -- 172.21.15.67:0/401360274 learned_addr learned my addr 172.21.15.67:0/401360274 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.490+0000 7f1392ffd700 1 -- 172.21.15.67:0/401360274 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1394108ff0 con 0x7f1394107df0 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.490+0000 7f1392ffd700 1 --2- 172.21.15.67:0/401360274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 0x7f1394108210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f13880098a0 tx=0x7f1388009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd4d7537db857b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.490+0000 7f1391ffb700 1 -- 172.21.15.67:0/401360274 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1388004160 con 0x7f1394107df0 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.490+0000 7f1391ffb700 1 -- 172.21.15.67:0/401360274 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1388004300 con 0x7f1394107df0 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.490+0000 7f1391ffb700 1 -- 172.21.15.67:0/401360274 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1388011af0 con 0x7f1394107df0 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.491+0000 7f1399369700 1 -- 172.21.15.67:0/401360274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 msgr2=0x7f1394108210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.491+0000 7f1399369700 1 --2- 172.21.15.67:0/401360274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 0x7f1394108210 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f13880098a0 tx=0x7f1388009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.491+0000 7f1399369700 1 -- 172.21.15.67:0/401360274 shutdown_connections 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.491+0000 7f1399369700 1 --2- 172.21.15.67:0/401360274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 0x7f1394108210 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.491+0000 7f1399369700 1 -- 172.21.15.67:0/401360274 >> 172.21.15.67:0/401360274 conn(0x7f1394103350 msgr2=0x7f13941057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:26.160 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.491+0000 7f1399369700 1 -- 172.21.15.67:0/401360274 shutdown_connections 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.491+0000 7f1399369700 1 -- 172.21.15.67:0/401360274 wait complete. 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.492+0000 7f1399369700 1 Processor -- start 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.492+0000 7f1399369700 1 -- start start 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.492+0000 7f1399369700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 0x7f139407d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.492+0000 7f1399369700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f139410c220 con 0x7f1394107df0 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.492+0000 7f1392ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 0x7f139407d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.492+0000 7f1392ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 0x7f139407d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33498/0 (socket says 172.21.15.67:33498) 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.492+0000 7f1392ffd700 1 -- 172.21.15.67:0/1279907978 learned_addr learned my addr 172.21.15.67:0/1279907978 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.492+0000 7f1392ffd700 1 -- 172.21.15.67:0/1279907978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139419fb70 con 0x7f1394107df0 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.493+0000 7f1392ffd700 1 --2- 172.21.15.67:0/1279907978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 0x7f139407d300 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1388009fb0 tx=0x7f1388011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.493+0000 7f137bfff700 1 -- 172.21.15.67:0/1279907978 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13880036a0 con 0x7f1394107df0 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.493+0000 7f1399369700 1 -- 172.21.15.67:0/1279907978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f139419fd40 con 0x7f1394107df0 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.493+0000 7f137bfff700 1 -- 172.21.15.67:0/1279907978 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f138801a430 con 0x7f1394107df0 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.493+0000 7f1399369700 1 -- 172.21.15.67:0/1279907978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13941a00e0 con 0x7f1394107df0 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.493+0000 7f137bfff700 1 -- 172.21.15.67:0/1279907978 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1388019550 con 0x7f1394107df0 2024-05-03T17:12:26.161 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.493+0000 7f137bfff700 1 -- 172.21.15.67:0/1279907978 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1388019be0 con 0x7f1394107df0 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.493+0000 7f137bfff700 1 -- 172.21.15.67:0/1279907978 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1388024730 con 0x7f1394107df0 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.494+0000 7f1399369700 1 -- 172.21.15.67:0/1279907978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1380005360 con 0x7f1394107df0 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.496+0000 7f137bfff700 1 -- 172.21.15.67:0/1279907978 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1388043e50 con 0x7f1394107df0 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.570+0000 7f1399369700 1 -- 172.21.15.67:0/1279907978 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1380005920 con 0x7f1394107df0 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.570+0000 7f137bfff700 1 -- 172.21.15.67:0/1279907978 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f1388032020 con 0x7f1394107df0 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.571+0000 7f1399369700 1 -- 172.21.15.67:0/1279907978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 msgr2=0x7f139407d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.571+0000 7f1399369700 1 --2- 172.21.15.67:0/1279907978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 0x7f139407d300 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1388009fb0 tx=0x7f1388011940 comp rx=0 tx=0).stop 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.571+0000 7f1399369700 1 -- 172.21.15.67:0/1279907978 shutdown_connections 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.571+0000 7f1399369700 1 --2- 172.21.15.67:0/1279907978 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1394107df0 0x7f139407d300 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.571+0000 7f1399369700 1 -- 172.21.15.67:0/1279907978 >> 172.21.15.67:0/1279907978 conn(0x7f1394103350 msgr2=0x7f1394103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.571+0000 7f1399369700 1 -- 172.21.15.67:0/1279907978 shutdown_connections 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:25.571+0000 7f1399369700 1 -- 172.21.15.67:0/1279907978 wait complete. 2024-05-03T17:12:26.162 INFO:teuthology.orchestra.run.smithi067.stdout:mgr not available, waiting (3/15)... 2024-05-03T17:12:26.453 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:26 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:26.005+0000 7f3341d12200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T17:12:26.453 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:26 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:26.229+0000 7f3341d12200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T17:12:26.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: Activating manager daemon x 2024-05-03T17:12:26.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: mgrmap e2: x(active, starting, since 0.0032863s) 2024-05-03T17:12:26.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: from='mgr.14100 172.21.15.67:0/579730608' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-03T17:12:26.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: from='mgr.14100 172.21.15.67:0/579730608' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-03T17:12:26.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: from='mgr.14100 172.21.15.67:0/579730608' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-03T17:12:26.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: from='mgr.14100 172.21.15.67:0/579730608' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:12:26.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: from='mgr.14100 172.21.15.67:0/579730608' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-03T17:12:26.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: Manager daemon x is now available 2024-05-03T17:12:26.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: from='mgr.14100 172.21.15.67:0/579730608' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-03T17:12:26.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: from='mgr.14100 172.21.15.67:0/579730608' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-03T17:12:26.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: from='mgr.14100 172.21.15.67:0/579730608' entity='mgr.x' 2024-05-03T17:12:26.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: from='mgr.14100 172.21.15.67:0/579730608' entity='mgr.x' 2024-05-03T17:12:26.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:26 smithi067 ceph-mon[29731]: from='mgr.14100 172.21.15.67:0/579730608' entity='mgr.x' 2024-05-03T17:12:28.604 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:28 smithi067 ceph-mon[29731]: mgrmap e3: x(active, since 1.008s) 2024-05-03T17:12:29.348 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:29 smithi067 ceph-mon[29731]: mgrmap e4: x(active, since 2s) 2024-05-03T17:12:29.348 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:29 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3058892159' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-03T17:12:29.654 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:29.654 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T17:12:29.654 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsid": "20c8e22e-0970-11ef-bc97-c7b262605968", 2024-05-03T17:12:29.654 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "health": { 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 0 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "a" 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-03T17:12:29.655 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:29.656 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-03T17:12:29.656 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-03T17:12:29.656 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:29.656 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-03T17:12:29.656 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:29.656 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-03T17:12:29.656 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-03T17:12:29.656 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-03T17:12:29.665 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-03T17:12:29.665 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-03T17:12:29.665 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-03T17:12:29.665 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:29.665 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-03T17:12:29.665 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-03T17:12:29.665 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-03T17:12:29.665 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-03T17:12:29.666 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "restful" 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ], 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "modified": "2024-05-03T17:12:11.709878+0000", 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout }, 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T17:12:29.667 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.715+0000 7f4911ce7700 1 Processor -- start 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.715+0000 7f4911ce7700 1 -- start start 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.716+0000 7f4911ce7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 0x7f490c10bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.716+0000 7f4911ce7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f490c07ce70 con 0x7f490c109990 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.716+0000 7f490b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 0x7f490c10bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.716+0000 7f490b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 0x7f490c10bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58996/0 (socket says 172.21.15.67:58996) 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.716+0000 7f490b7fe700 1 -- 172.21.15.67:0/4206632847 learned_addr learned my addr 172.21.15.67:0/4206632847 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.716+0000 7f490b7fe700 1 -- 172.21.15.67:0/4206632847 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490c10c2c0 con 0x7f490c109990 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.716+0000 7f490b7fe700 1 --2- 172.21.15.67:0/4206632847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 0x7f490c10bd80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f48f40098a0 tx=0x7f48f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab46e7094abb207a server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.717+0000 7f490a7fc700 1 -- 172.21.15.67:0/4206632847 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48f4004160 con 0x7f490c109990 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.717+0000 7f490a7fc700 1 -- 172.21.15.67:0/4206632847 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f48f4004300 con 0x7f490c109990 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.717+0000 7f490a7fc700 1 -- 172.21.15.67:0/4206632847 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48f4011af0 con 0x7f490c109990 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.717+0000 7f4911ce7700 1 -- 172.21.15.67:0/4206632847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 msgr2=0x7f490c10bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.717+0000 7f4911ce7700 1 --2- 172.21.15.67:0/4206632847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 0x7f490c10bd80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f48f40098a0 tx=0x7f48f4009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.717+0000 7f4911ce7700 1 -- 172.21.15.67:0/4206632847 shutdown_connections 2024-05-03T17:12:29.668 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.717+0000 7f4911ce7700 1 --2- 172.21.15.67:0/4206632847 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 0x7f490c10bd80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.717+0000 7f4911ce7700 1 -- 172.21.15.67:0/4206632847 >> 172.21.15.67:0/4206632847 conn(0x7f490c103380 msgr2=0x7f490c1057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.717+0000 7f4911ce7700 1 -- 172.21.15.67:0/4206632847 shutdown_connections 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.717+0000 7f4911ce7700 1 -- 172.21.15.67:0/4206632847 wait complete. 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.718+0000 7f4911ce7700 1 Processor -- start 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.718+0000 7f4911ce7700 1 -- start start 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.718+0000 7f4911ce7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 0x7f490c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.718+0000 7f4911ce7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f490c10d1c0 con 0x7f490c109990 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.719+0000 7f490b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 0x7f490c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.719+0000 7f490b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 0x7f490c19cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58998/0 (socket says 172.21.15.67:58998) 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.719+0000 7f490b7fe700 1 -- 172.21.15.67:0/3058892159 learned_addr learned my addr 172.21.15.67:0/3058892159 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.719+0000 7f490b7fe700 1 -- 172.21.15.67:0/3058892159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f490c19d2a0 con 0x7f490c109990 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.719+0000 7f490b7fe700 1 --2- 172.21.15.67:0/3058892159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 0x7f490c19cd60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f48f4011ac0 tx=0x7f48f4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.719+0000 7f4908ff9700 1 -- 172.21.15.67:0/3058892159 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48f4003b20 con 0x7f490c109990 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.719+0000 7f4908ff9700 1 -- 172.21.15.67:0/3058892159 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f48f401a8c0 con 0x7f490c109990 2024-05-03T17:12:29.669 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.719+0000 7f4911ce7700 1 -- 172.21.15.67:0/3058892159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f490c19d530 con 0x7f490c109990 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.719+0000 7f4908ff9700 1 -- 172.21.15.67:0/3058892159 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f48f40248d0 con 0x7f490c109990 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.719+0000 7f4911ce7700 1 -- 172.21.15.67:0/3058892159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f490c1a0220 con 0x7f490c109990 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.720+0000 7f4908ff9700 1 -- 172.21.15.67:0/3058892159 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f48f4022020 con 0x7f490c109990 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.720+0000 7f4908ff9700 1 --2- 172.21.15.67:0/3058892159 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f48f803e420 0x7f48f80408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.720+0000 7f4908ff9700 1 -- 172.21.15.67:0/3058892159 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f48f40537a0 con 0x7f490c109990 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.720+0000 7f490affd700 1 --2- 172.21.15.67:0/3058892159 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f48f803e420 0x7f48f80408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.720+0000 7f4911ce7700 1 -- 172.21.15.67:0/3058892159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f490c064b00 con 0x7f490c109990 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.721+0000 7f490affd700 1 --2- 172.21.15.67:0/3058892159 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f48f803e420 0x7f48f80408e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f48fc006fd0 tx=0x7f48fc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:28.726+0000 7f4908ff9700 1 -- 172.21.15.67:0/3058892159 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f48f400e260 con 0x7f490c109990 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:29.044+0000 7f4911ce7700 1 -- 172.21.15.67:0/3058892159 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f490c1a06e0 con 0x7f490c109990 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:29.045+0000 7f4908ff9700 1 -- 172.21.15.67:0/3058892159 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f48f4053500 con 0x7f490c109990 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:29.048+0000 7f4911ce7700 1 -- 172.21.15.67:0/3058892159 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f48f803e420 msgr2=0x7f48f80408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:29.670 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:29.048+0000 7f4911ce7700 1 --2- 172.21.15.67:0/3058892159 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f48f803e420 0x7f48f80408e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f48fc006fd0 tx=0x7f48fc006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:29.671 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:29.048+0000 7f4911ce7700 1 -- 172.21.15.67:0/3058892159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 msgr2=0x7f490c19cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:29.671 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:29.048+0000 7f4911ce7700 1 --2- 172.21.15.67:0/3058892159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 0x7f490c19cd60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f48f4011ac0 tx=0x7f48f4011ba0 comp rx=0 tx=0).stop 2024-05-03T17:12:29.671 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:29.048+0000 7f4911ce7700 1 -- 172.21.15.67:0/3058892159 shutdown_connections 2024-05-03T17:12:29.671 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:29.048+0000 7f4911ce7700 1 --2- 172.21.15.67:0/3058892159 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f48f803e420 0x7f48f80408e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:29.671 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:29.048+0000 7f4911ce7700 1 --2- 172.21.15.67:0/3058892159 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f490c109990 0x7f490c19cd60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:29.671 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:29.048+0000 7f4911ce7700 1 -- 172.21.15.67:0/3058892159 >> 172.21.15.67:0/3058892159 conn(0x7f490c103380 msgr2=0x7f490c103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:29.671 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:29.048+0000 7f4911ce7700 1 -- 172.21.15.67:0/3058892159 shutdown_connections 2024-05-03T17:12:29.671 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:29.048+0000 7f4911ce7700 1 -- 172.21.15.67:0/3058892159 wait complete. 2024-05-03T17:12:29.671 INFO:teuthology.orchestra.run.smithi067.stdout:mgr is available 2024-05-03T17:12:30.553 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:30 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1339225311' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-03T17:12:30.839 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:30.839 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [global] 2024-05-03T17:12:30.839 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout fsid = 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:12:30.839 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-03T17:12:30.839 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-03T17:12:30.839 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-03T17:12:30.839 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-03T17:12:30.839 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-03T17:12:30.839 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:30.839 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout [osd] 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.024+0000 7fcc6c0fe700 1 Processor -- start 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.024+0000 7fcc6c0fe700 1 -- start start 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.024+0000 7fcc6c0fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 0x7fcc641081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.025+0000 7fcc6c0fe700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcc641087b0 con 0x7fcc64107dc0 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.025+0000 7fcc69e9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 0x7fcc641081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:30.840 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.025+0000 7fcc69e9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 0x7fcc641081e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59008/0 (socket says 172.21.15.67:59008) 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.025+0000 7fcc69e9a700 1 -- 172.21.15.67:0/626890754 learned_addr learned my addr 172.21.15.67:0/626890754 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.025+0000 7fcc69e9a700 1 -- 172.21.15.67:0/626890754 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc64108fc0 con 0x7fcc64107dc0 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.025+0000 7fcc69e9a700 1 --2- 172.21.15.67:0/626890754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 0x7fcc641081e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcc54009e80 tx=0x7fcc5400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c88989805964ce server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.025+0000 7fcc68e98700 1 -- 172.21.15.67:0/626890754 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc54017070 con 0x7fcc64107dc0 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.025+0000 7fcc68e98700 1 -- 172.21.15.67:0/626890754 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcc5400b8c0 con 0x7fcc64107dc0 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.025+0000 7fcc68e98700 1 -- 172.21.15.67:0/626890754 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc54004490 con 0x7fcc64107dc0 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.026+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/626890754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 msgr2=0x7fcc641081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.026+0000 7fcc6c0fe700 1 --2- 172.21.15.67:0/626890754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 0x7fcc641081e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fcc54009e80 tx=0x7fcc5400b330 comp rx=0 tx=0).stop 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.026+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/626890754 shutdown_connections 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.026+0000 7fcc6c0fe700 1 --2- 172.21.15.67:0/626890754 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 0x7fcc641081e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.026+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/626890754 >> 172.21.15.67:0/626890754 conn(0x7fcc64103380 msgr2=0x7fcc641057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.026+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/626890754 shutdown_connections 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.026+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/626890754 wait complete. 2024-05-03T17:12:30.841 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.027+0000 7fcc6c0fe700 1 Processor -- start 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.027+0000 7fcc6c0fe700 1 -- start start 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.027+0000 7fcc6c0fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 0x7fcc64192eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.027+0000 7fcc6c0fe700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcc6410c1f0 con 0x7fcc64107dc0 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.027+0000 7fcc69e9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 0x7fcc64192eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.027+0000 7fcc69e9a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 0x7fcc64192eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59024/0 (socket says 172.21.15.67:59024) 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.027+0000 7fcc69e9a700 1 -- 172.21.15.67:0/1339225311 learned_addr learned my addr 172.21.15.67:0/1339225311 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.027+0000 7fcc69e9a700 1 -- 172.21.15.67:0/1339225311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc641933f0 con 0x7fcc64107dc0 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.027+0000 7fcc69e9a700 1 --2- 172.21.15.67:0/1339225311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 0x7fcc64192eb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcc54004a60 tx=0x7fcc5400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.028+0000 7fcc62ffd700 1 -- 172.21.15.67:0/1339225311 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc54020020 con 0x7fcc64107dc0 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.028+0000 7fcc62ffd700 1 -- 172.21.15.67:0/1339225311 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcc54003c60 con 0x7fcc64107dc0 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.028+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/1339225311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc641915e0 con 0x7fcc64107dc0 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.028+0000 7fcc62ffd700 1 -- 172.21.15.67:0/1339225311 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcc54017040 con 0x7fcc64107dc0 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.028+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/1339225311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc64191b20 con 0x7fcc64107dc0 2024-05-03T17:12:30.842 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.029+0000 7fcc62ffd700 1 -- 172.21.15.67:0/1339225311 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7fcc54023020 con 0x7fcc64107dc0 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.029+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/1339225311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc6410c370 con 0x7fcc64107dc0 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.029+0000 7fcc62ffd700 1 --2- 172.21.15.67:0/1339225311 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7fcc4c03e470 0x7fcc4c040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.029+0000 7fcc62ffd700 1 -- 172.21.15.67:0/1339225311 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcc5400f050 con 0x7fcc64107dc0 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.029+0000 7fcc69699700 1 --2- 172.21.15.67:0/1339225311 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7fcc4c03e470 0x7fcc4c040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.029+0000 7fcc69699700 1 --2- 172.21.15.67:0/1339225311 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7fcc4c03e470 0x7fcc4c040930 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcc58006fd0 tx=0x7fcc58006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.034+0000 7fcc62ffd700 1 -- 172.21.15.67:0/1339225311 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fcc5402a020 con 0x7fcc64107dc0 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.246+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/1339225311 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fcc64196d00 con 0x7fcc64107dc0 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.246+0000 7fcc62ffd700 1 -- 172.21.15.67:0/1339225311 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7fcc54003e10 con 0x7fcc64107dc0 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.249+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/1339225311 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7fcc4c03e470 msgr2=0x7fcc4c040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.249+0000 7fcc6c0fe700 1 --2- 172.21.15.67:0/1339225311 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7fcc4c03e470 0x7fcc4c040930 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcc58006fd0 tx=0x7fcc58006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.249+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/1339225311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 msgr2=0x7fcc64192eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.249+0000 7fcc6c0fe700 1 --2- 172.21.15.67:0/1339225311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 0x7fcc64192eb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcc54004a60 tx=0x7fcc5400bd50 comp rx=0 tx=0).stop 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.249+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/1339225311 shutdown_connections 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.249+0000 7fcc6c0fe700 1 --2- 172.21.15.67:0/1339225311 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7fcc4c03e470 0x7fcc4c040930 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:30.843 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.249+0000 7fcc6c0fe700 1 --2- 172.21.15.67:0/1339225311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc64107dc0 0x7fcc64192eb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:30.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.249+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/1339225311 >> 172.21.15.67:0/1339225311 conn(0x7fcc64103380 msgr2=0x7fcc64104200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:30.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.249+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/1339225311 shutdown_connections 2024-05-03T17:12:30.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:30.249+0000 7fcc6c0fe700 1 -- 172.21.15.67:0/1339225311 wait complete. 2024-05-03T17:12:30.844 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling cephadm module... 2024-05-03T17:12:31.853 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:31 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2026066781' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-03T17:12:31.853 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:31 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: ignoring --setuser ceph since I am not root 2024-05-03T17:12:31.853 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:31 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: ignoring --setgroup ceph since I am not root 2024-05-03T17:12:31.853 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:31 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:31.761+0000 7f392e696200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T17:12:32.144 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:31 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:31.951+0000 7f392e696200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T17:12:32.145 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:32 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:32.057+0000 7f392e696200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T17:12:32.152 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.253+0000 7f4001c60700 1 Processor -- start 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.253+0000 7f4001c60700 1 -- start start 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.253+0000 7f4001c60700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 0x7f3ffc1098a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.254+0000 7f4001c60700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc109e70 con 0x7f3ffc109480 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.254+0000 7f3ffb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 0x7f3ffc1098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.254+0000 7f3ffb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 0x7f3ffc1098a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59028/0 (socket says 172.21.15.67:59028) 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.254+0000 7f3ffb7fe700 1 -- 172.21.15.67:0/1742640598 learned_addr learned my addr 172.21.15.67:0/1742640598 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.254+0000 7f3ffb7fe700 1 -- 172.21.15.67:0/1742640598 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ffc108050 con 0x7f3ffc109480 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.254+0000 7f3ffb7fe700 1 --2- 172.21.15.67:0/1742640598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 0x7f3ffc1098a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3fec009e80 tx=0x7f3fec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f230a076642ed051 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.254+0000 7f3ffa7fc700 1 -- 172.21.15.67:0/1742640598 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3fec017070 con 0x7f3ffc109480 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.254+0000 7f3ffa7fc700 1 -- 172.21.15.67:0/1742640598 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3fec00b8c0 con 0x7f3ffc109480 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.254+0000 7f3ffa7fc700 1 -- 172.21.15.67:0/1742640598 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3fec004490 con 0x7f3ffc109480 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.255+0000 7f4001c60700 1 -- 172.21.15.67:0/1742640598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 msgr2=0x7f3ffc1098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.255+0000 7f4001c60700 1 --2- 172.21.15.67:0/1742640598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 0x7f3ffc1098a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3fec009e80 tx=0x7f3fec00b330 comp rx=0 tx=0).stop 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.255+0000 7f4001c60700 1 -- 172.21.15.67:0/1742640598 shutdown_connections 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.255+0000 7f4001c60700 1 --2- 172.21.15.67:0/1742640598 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 0x7f3ffc1098a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:32.153 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.255+0000 7f4001c60700 1 -- 172.21.15.67:0/1742640598 >> 172.21.15.67:0/1742640598 conn(0x7f3ffc102b20 msgr2=0x7f3ffc104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.255+0000 7f4001c60700 1 -- 172.21.15.67:0/1742640598 shutdown_connections 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.255+0000 7f4001c60700 1 -- 172.21.15.67:0/1742640598 wait complete. 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.256+0000 7f4001c60700 1 Processor -- start 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.256+0000 7f4001c60700 1 -- start start 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.256+0000 7f4001c60700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 0x7f3ffc19ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.256+0000 7f4001c60700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3ffc10c290 con 0x7f3ffc109480 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.256+0000 7f3ffb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 0x7f3ffc19ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.256+0000 7f3ffb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 0x7f3ffc19ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59034/0 (socket says 172.21.15.67:59034) 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.256+0000 7f3ffb7fe700 1 -- 172.21.15.67:0/2026066781 learned_addr learned my addr 172.21.15.67:0/2026066781 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.256+0000 7f3ffb7fe700 1 -- 172.21.15.67:0/2026066781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ffc19d3a0 con 0x7f3ffc109480 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.256+0000 7f3ffb7fe700 1 --2- 172.21.15.67:0/2026066781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 0x7f3ffc19ce60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00b7b0 tx=0x7f3fec00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.257+0000 7f3ff8ff9700 1 -- 172.21.15.67:0/2026066781 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3fec020020 con 0x7f3ffc109480 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.257+0000 7f4001c60700 1 -- 172.21.15.67:0/2026066781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ffc19d630 con 0x7f3ffc109480 2024-05-03T17:12:32.154 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.257+0000 7f3ff8ff9700 1 -- 172.21.15.67:0/2026066781 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3fec012430 con 0x7f3ffc109480 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.257+0000 7f3ff8ff9700 1 -- 172.21.15.67:0/2026066781 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3fec017040 con 0x7f3ffc109480 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.257+0000 7f4001c60700 1 -- 172.21.15.67:0/2026066781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ffc1a0320 con 0x7f3ffc109480 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.258+0000 7f3ff8ff9700 1 -- 172.21.15.67:0/2026066781 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f3fec023020 con 0x7f3ffc109480 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.258+0000 7f4001c60700 1 -- 172.21.15.67:0/2026066781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ffc064b00 con 0x7f3ffc109480 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.258+0000 7f3ff8ff9700 1 --2- 172.21.15.67:0/2026066781 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f3fe803e3d0 0x7f3fe8040890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.258+0000 7f3ff8ff9700 1 -- 172.21.15.67:0/2026066781 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3fec00f050 con 0x7f3ffc109480 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.258+0000 7f3ffaffd700 1 --2- 172.21.15.67:0/2026066781 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f3fe803e3d0 0x7f3fe8040890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.259+0000 7f3ffaffd700 1 --2- 172.21.15.67:0/2026066781 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f3fe803e3d0 0x7f3fe8040890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4006fd0 tx=0x7f3fe4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.263+0000 7f3ff8ff9700 1 -- 172.21.15.67:0/2026066781 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f3fec02a020 con 0x7f3ffc109480 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.538+0000 7f4001c60700 1 -- 172.21.15.67:0/2026066781 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f3ffc02e850 con 0x7f3ffc109480 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.590+0000 7f3ff8ff9700 1 -- 172.21.15.67:0/2026066781 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f3fec02a200 con 0x7f3ffc109480 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.591+0000 7f3ff8ff9700 1 -- 172.21.15.67:0/2026066781 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f3fec01bc90 con 0x7f3ffc109480 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.593+0000 7f4001c60700 1 -- 172.21.15.67:0/2026066781 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f3fe803e3d0 msgr2=0x7f3fe8040890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.593+0000 7f4001c60700 1 --2- 172.21.15.67:0/2026066781 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f3fe803e3d0 0x7f3fe8040890 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3fe4006fd0 tx=0x7f3fe4006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:32.155 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.593+0000 7f4001c60700 1 -- 172.21.15.67:0/2026066781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 msgr2=0x7f3ffc19ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.593+0000 7f4001c60700 1 --2- 172.21.15.67:0/2026066781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 0x7f3ffc19ce60 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3fec00b7b0 tx=0x7f3fec00be20 comp rx=0 tx=0).stop 2024-05-03T17:12:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.593+0000 7f4001c60700 1 -- 172.21.15.67:0/2026066781 shutdown_connections 2024-05-03T17:12:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.593+0000 7f4001c60700 1 --2- 172.21.15.67:0/2026066781 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f3fe803e3d0 0x7f3fe8040890 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.593+0000 7f4001c60700 1 --2- 172.21.15.67:0/2026066781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3ffc109480 0x7f3ffc19ce60 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.593+0000 7f4001c60700 1 -- 172.21.15.67:0/2026066781 >> 172.21.15.67:0/2026066781 conn(0x7f3ffc102b20 msgr2=0x7f3ffc103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.593+0000 7f4001c60700 1 -- 172.21.15.67:0/2026066781 shutdown_connections 2024-05-03T17:12:32.156 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:31.593+0000 7f4001c60700 1 -- 172.21.15.67:0/2026066781 wait complete. 2024-05-03T17:12:32.404 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:32 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:32.381+0000 7f392e696200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T17:12:32.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:32 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2026066781' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-03T17:12:32.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:32 smithi067 ceph-mon[29731]: mgrmap e5: x(active, since 5s) 2024-05-03T17:12:32.704 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:32 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:32.471+0000 7f392e696200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T17:12:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T17:12:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-03T17:12:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-03T17:12:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-03T17:12:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-03T17:12:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T17:12:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.606+0000 7f1ebc2ec700 1 Processor -- start 2024-05-03T17:12:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.606+0000 7f1ebc2ec700 1 -- start start 2024-05-03T17:12:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.606+0000 7f1ebc2ec700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 0x7f1eb41075d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.606+0000 7f1ebc2ec700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb4107ba0 con 0x7f1eb41071b0 2024-05-03T17:12:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.607+0000 7f1eba088700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 0x7f1eb41075d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:33.477 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.607+0000 7f1eba088700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 0x7f1eb41075d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59062/0 (socket says 172.21.15.67:59062) 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.607+0000 7f1eba088700 1 -- 172.21.15.67:0/2374236551 learned_addr learned my addr 172.21.15.67:0/2374236551 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.607+0000 7f1eba088700 1 -- 172.21.15.67:0/2374236551 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb410a140 con 0x7f1eb41071b0 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.607+0000 7f1eba088700 1 --2- 172.21.15.67:0/2374236551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 0x7f1eb41075d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00098a0 tx=0x7f1eb0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea0be6b7bea6b6b server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.607+0000 7f1eb9086700 1 -- 172.21.15.67:0/2374236551 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1eb0004160 con 0x7f1eb41071b0 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.607+0000 7f1eb9086700 1 -- 172.21.15.67:0/2374236551 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1eb0004300 con 0x7f1eb41071b0 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.607+0000 7f1eb9086700 1 -- 172.21.15.67:0/2374236551 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1eb0011af0 con 0x7f1eb41071b0 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.608+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/2374236551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 msgr2=0x7f1eb41075d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.608+0000 7f1ebc2ec700 1 --2- 172.21.15.67:0/2374236551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 0x7f1eb41075d0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f1eb00098a0 tx=0x7f1eb0009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.608+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/2374236551 shutdown_connections 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.608+0000 7f1ebc2ec700 1 --2- 172.21.15.67:0/2374236551 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 0x7f1eb41075d0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.608+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/2374236551 >> 172.21.15.67:0/2374236551 conn(0x7f1eb4102b00 msgr2=0x7f1eb4104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.608+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/2374236551 shutdown_connections 2024-05-03T17:12:33.478 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.608+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/2374236551 wait complete. 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.609+0000 7f1ebc2ec700 1 Processor -- start 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.609+0000 7f1ebc2ec700 1 -- start start 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.609+0000 7f1ebc2ec700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 0x7f1eb41099c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.609+0000 7f1ebc2ec700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb410d370 con 0x7f1eb41071b0 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.609+0000 7f1eba088700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 0x7f1eb41099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.609+0000 7f1eba088700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 0x7f1eb41099c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59064/0 (socket says 172.21.15.67:59064) 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.609+0000 7f1eba088700 1 -- 172.21.15.67:0/1710131823 learned_addr learned my addr 172.21.15.67:0/1710131823 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.609+0000 7f1eba088700 1 -- 172.21.15.67:0/1710131823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb4109f00 con 0x7f1eb41071b0 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.610+0000 7f1eba088700 1 --2- 172.21.15.67:0/1710131823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 0x7f1eb41099c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0000c00 tx=0x7f1eb0011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.610+0000 7f1eab7fe700 1 -- 172.21.15.67:0/1710131823 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1eb0003b20 con 0x7f1eb41071b0 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.610+0000 7f1eab7fe700 1 -- 172.21.15.67:0/1710131823 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1eb001a8c0 con 0x7f1eb41071b0 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.610+0000 7f1eab7fe700 1 -- 172.21.15.67:0/1710131823 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1eb0024940 con 0x7f1eb41071b0 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.610+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/1710131823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb41080f0 con 0x7f1eb41071b0 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.610+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/1710131823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eb4108630 con 0x7f1eb41071b0 2024-05-03T17:12:33.479 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.611+0000 7f1eab7fe700 1 -- 172.21.15.67:0/1710131823 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f1eb0022020 con 0x7f1eb41071b0 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.611+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/1710131823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eb4064b00 con 0x7f1eb41071b0 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.611+0000 7f1eab7fe700 1 --2- 172.21.15.67:0/1710131823 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f1ea003e470 0x7f1ea0040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.611+0000 7f1eab7fe700 1 -- 172.21.15.67:0/1710131823 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1eb0053560 con 0x7f1eb41071b0 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.611+0000 7f1eb9887700 1 -- 172.21.15.67:0/1710131823 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f1ea003e470 msgr2=0x7f1ea0040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1345903711 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.611+0000 7f1eb9887700 1 --2- 172.21.15.67:0/1710131823 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f1ea003e470 0x7f1ea0040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.616+0000 7f1eab7fe700 1 -- 172.21.15.67:0/1710131823 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f1eb0018dc0 con 0x7f1eb41071b0 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.812+0000 7f1eb9887700 1 -- 172.21.15.67:0/1710131823 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f1ea003e470 msgr2=0x7f1ea0040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1345903711 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.812+0000 7f1eb9887700 1 --2- 172.21.15.67:0/1710131823 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f1ea003e470 0x7f1ea0040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.933+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/1710131823 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f1eb4108f60 con 0x7f1eb41071b0 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.934+0000 7f1eab7fe700 1 -- 172.21.15.67:0/1710131823 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f1eb00532c0 con 0x7f1eb41071b0 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.936+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/1710131823 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f1ea003e470 msgr2=0x7f1ea0040930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.936+0000 7f1ebc2ec700 1 --2- 172.21.15.67:0/1710131823 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f1ea003e470 0x7f1ea0040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.936+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/1710131823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 msgr2=0x7f1eb41099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:33.480 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.936+0000 7f1ebc2ec700 1 --2- 172.21.15.67:0/1710131823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 0x7f1eb41099c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1eb0000c00 tx=0x7f1eb0011b70 comp rx=0 tx=0).stop 2024-05-03T17:12:33.481 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.936+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/1710131823 shutdown_connections 2024-05-03T17:12:33.481 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.936+0000 7f1ebc2ec700 1 --2- 172.21.15.67:0/1710131823 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f1ea003e470 0x7f1ea0040930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:33.481 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.936+0000 7f1ebc2ec700 1 --2- 172.21.15.67:0/1710131823 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1eb41071b0 0x7f1eb41099c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:33.481 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.936+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/1710131823 >> 172.21.15.67:0/1710131823 conn(0x7f1eb4102b00 msgr2=0x7f1eb4103590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:33.481 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.936+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/1710131823 shutdown_connections 2024-05-03T17:12:33.481 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:32.936+0000 7f1ebc2ec700 1 -- 172.21.15.67:0/1710131823 wait complete. 2024-05-03T17:12:33.481 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for the mgr to restart... 2024-05-03T17:12:33.481 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr epoch 5... 2024-05-03T17:12:33.903 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:33 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1710131823' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-03T17:12:33.904 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:33 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:33.721+0000 7f392e696200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T17:12:34.203 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:33 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:33.902+0000 7f392e696200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T17:12:34.203 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:34 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:34.080+0000 7f392e696200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T17:12:34.686 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:34 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:34.380+0000 7f392e696200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T17:12:34.686 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:34 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:34.475+0000 7f392e696200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T17:12:34.686 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:34 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:34.684+0000 7f392e696200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T17:12:34.953 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:34 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:34.883+0000 7f392e696200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T17:12:35.703 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:35 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:35.506+0000 7f392e696200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T17:12:36.176 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:35 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:35.734+0000 7f392e696200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-03T17:12:36.453 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:36 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:36.175+0000 7f392e696200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T17:12:36.453 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:36 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:36.274+0000 7f392e696200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T17:12:36.949 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:36 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:36.669+0000 7f392e696200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-03T17:12:37.203 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:37 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:37.038+0000 7f392e696200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T17:12:37.203 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:37 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:37.133+0000 7f392e696200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T17:12:37.703 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:37 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:37.383+0000 7f392e696200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T17:12:37.703 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:37 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:37.520+0000 7f392e696200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T17:12:38.203 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:37 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:37.837+0000 7f392e696200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T17:12:38.203 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:37 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:37.956+0000 7f392e696200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T17:12:38.203 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:38 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:38.050+0000 7f392e696200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T17:12:38.951 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:38 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:38.672+0000 7f392e696200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T17:12:38.952 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:38 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:38.896+0000 7f392e696200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T17:12:39.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: Active manager daemon x restarted 2024-05-03T17:12:39.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: Activating manager daemon x 2024-05-03T17:12:39.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: osdmap e2: 0 total, 0 up, 0 in 2024-05-03T17:12:39.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: mgrmap e6: x(active, starting, since 0.00350428s) 2024-05-03T17:12:39.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:12:39.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-03T17:12:39.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-03T17:12:39.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-03T17:12:39.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-03T17:12:39.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: Manager daemon x is now available 2024-05-03T17:12:39.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:39.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:39.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:12:39.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:38 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:12:40.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:39 smithi067 ceph-mon[29731]: Found migration_current of "None". Setting to last migration. 2024-05-03T17:12:40.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:39 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-03T17:12:40.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:39 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-03T17:12:40.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:39 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:40.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:39 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:40.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:39 smithi067 ceph-mon[29731]: mgrmap e7: x(active, since 1.0058s) 2024-05-03T17:12:40.526 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T17:12:40.527 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-03T17:12:40.527 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-03T17:12:40.527 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T17:12:40.527 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.938+0000 7f98bd106700 1 Processor -- start 2024-05-03T17:12:40.527 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.938+0000 7f98bd106700 1 -- start start 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.938+0000 7f98bd106700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 0x7f98b810bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.938+0000 7f98bd106700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f98b807ced0 con 0x7f98b8109930 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.938+0000 7f98b6d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 0x7f98b810bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.938+0000 7f98b6d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 0x7f98b810bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59066/0 (socket says 172.21.15.67:59066) 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.938+0000 7f98b6d9d700 1 -- 172.21.15.67:0/962602058 learned_addr learned my addr 172.21.15.67:0/962602058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.939+0000 7f98b6d9d700 1 -- 172.21.15.67:0/962602058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b810c260 con 0x7f98b8109930 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.939+0000 7f98b6d9d700 1 --2- 172.21.15.67:0/962602058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 0x7f98b810bd20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0098a0 tx=0x7f98ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3f4d0b38bc4933 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.939+0000 7f98b5d9b700 1 -- 172.21.15.67:0/962602058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98ac004160 con 0x7f98b8109930 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.939+0000 7f98b5d9b700 1 -- 172.21.15.67:0/962602058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f98ac004300 con 0x7f98b8109930 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.939+0000 7f98b5d9b700 1 -- 172.21.15.67:0/962602058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98ac011af0 con 0x7f98b8109930 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.940+0000 7f98bd106700 1 -- 172.21.15.67:0/962602058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 msgr2=0x7f98b810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.940+0000 7f98bd106700 1 --2- 172.21.15.67:0/962602058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 0x7f98b810bd20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0098a0 tx=0x7f98ac009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:40.528 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.940+0000 7f98bd106700 1 -- 172.21.15.67:0/962602058 shutdown_connections 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.940+0000 7f98bd106700 1 --2- 172.21.15.67:0/962602058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 0x7f98b810bd20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.940+0000 7f98bd106700 1 -- 172.21.15.67:0/962602058 >> 172.21.15.67:0/962602058 conn(0x7f98b8103380 msgr2=0x7f98b81057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.940+0000 7f98bd106700 1 -- 172.21.15.67:0/962602058 shutdown_connections 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.940+0000 7f98bd106700 1 -- 172.21.15.67:0/962602058 wait complete. 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.941+0000 7f98bd106700 1 Processor -- start 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.941+0000 7f98bd106700 1 -- start start 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.941+0000 7f98bd106700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 0x7f98b819cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.941+0000 7f98bd106700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f98b810d1c0 con 0x7f98b8109930 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.941+0000 7f98b6d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 0x7f98b819cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.941+0000 7f98b6d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 0x7f98b819cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59072/0 (socket says 172.21.15.67:59072) 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.941+0000 7f98b6d9d700 1 -- 172.21.15.67:0/4061868281 learned_addr learned my addr 172.21.15.67:0/4061868281 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.942+0000 7f98b6d9d700 1 -- 172.21.15.67:0/4061868281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98b819d270 con 0x7f98b8109930 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.942+0000 7f98b6d9d700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 0x7f98b819cd30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f98ac003f40 tx=0x7f98ac011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:40.529 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.942+0000 7f989ffff700 1 -- 172.21.15.67:0/4061868281 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98ac0038b0 con 0x7f98b8109930 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.942+0000 7f98bd106700 1 -- 172.21.15.67:0/4061868281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98b819d500 con 0x7f98b8109930 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.942+0000 7f989ffff700 1 -- 172.21.15.67:0/4061868281 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f98ac003a50 con 0x7f98b8109930 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.943+0000 7f989ffff700 1 -- 172.21.15.67:0/4061868281 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98ac01a5c0 con 0x7f98b8109930 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.943+0000 7f98bd106700 1 -- 172.21.15.67:0/4061868281 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98b81a01f0 con 0x7f98b8109930 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.943+0000 7f989ffff700 1 -- 172.21.15.67:0/4061868281 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f98ac01a870 con 0x7f98b8109930 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.943+0000 7f989ffff700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 0x7f98a0040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.943+0000 7f989ffff700 1 -- 172.21.15.67:0/4061868281 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f98ac055d30 con 0x7f98b8109930 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.943+0000 7f98b659c700 1 -- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 msgr2=0x7f98a0040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1345903711 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.943+0000 7f98b659c700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 0x7f98a0040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:33.943+0000 7f98bd106700 1 -- 172.21.15.67:0/4061868281 --> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f98b806a5e0 con 0x7f98a003e470 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:34.144+0000 7f98b659c700 1 -- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 msgr2=0x7f98a0040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1345903711 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:34.144+0000 7f98b659c700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 0x7f98a0040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-03T17:12:40.530 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:34.545+0000 7f98b659c700 1 -- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 msgr2=0x7f98a0040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1345903711 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:34.545+0000 7f98b659c700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 0x7f98a0040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:35.346+0000 7f98b659c700 1 -- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 msgr2=0x7f98a0040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1345903711 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:35.346+0000 7f98b659c700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 0x7f98a0040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:36.947+0000 7f98b659c700 1 -- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 msgr2=0x7f98a0040930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/1345903711 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:36.947+0000 7f98b659c700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 0x7f98a0040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:38.900+0000 7f989ffff700 1 -- 172.21.15.67:0/4061868281 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mgrmap(e 6) ==== 49989+0+0 (secure 0 0 0) 0x7f98ac019a20 con 0x7f98b8109930 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:38.900+0000 7f989ffff700 1 -- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 msgr2=0x7f98a0040930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:38.900+0000 7f989ffff700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 0x7f98a0040930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.901+0000 7f989ffff700 1 -- 172.21.15.67:0/4061868281 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7f98ac03b1e0 con 0x7f98b8109930 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.901+0000 7f989ffff700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f98a0041b70 0x7f98a0043f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.901+0000 7f989ffff700 1 -- 172.21.15.67:0/4061868281 --> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f98b806a5e0 con 0x7f98a0041b70 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.901+0000 7f98b659c700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f98a0041b70 0x7f98a0043f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.902+0000 7f98b659c700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f98a0041b70 0x7f98a0043f60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f98a8003a10 tx=0x7f98a80092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:40.531 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.903+0000 7f989ffff700 1 -- 172.21.15.67:0/4061868281 <== mgr.14120 v2:172.21.15.67:6800/3335149382 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f98b806a5e0 con 0x7f98a0041b70 2024-05-03T17:12:40.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.911+0000 7f98bd106700 1 -- 172.21.15.67:0/4061868281 --> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f98b8104600 con 0x7f98a0041b70 2024-05-03T17:12:40.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.911+0000 7f989ffff700 1 -- 172.21.15.67:0/4061868281 <== mgr.14120 v2:172.21.15.67:6800/3335149382 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f98b8104600 con 0x7f98a0041b70 2024-05-03T17:12:40.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.912+0000 7f98bd106700 1 -- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f98a0041b70 msgr2=0x7f98a0043f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:40.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.912+0000 7f98bd106700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f98a0041b70 0x7f98a0043f60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f98a8003a10 tx=0x7f98a80092b0 comp rx=0 tx=0).stop 2024-05-03T17:12:40.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.912+0000 7f98bd106700 1 -- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 msgr2=0x7f98b819cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:40.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.912+0000 7f98bd106700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 0x7f98b819cd30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f98ac003f40 tx=0x7f98ac011890 comp rx=0 tx=0).stop 2024-05-03T17:12:40.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.912+0000 7f98bd106700 1 -- 172.21.15.67:0/4061868281 shutdown_connections 2024-05-03T17:12:40.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.912+0000 7f98bd106700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f98a0041b70 0x7f98a0043f60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:40.532 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.912+0000 7f98bd106700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f98b8109930 0x7f98b819cd30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:40.533 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.912+0000 7f98bd106700 1 --2- 172.21.15.67:0/4061868281 >> [v2:172.21.15.67:6800/1345903711,v1:172.21.15.67:6801/1345903711] conn(0x7f98a003e470 0x7f98a0040930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:40.533 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.912+0000 7f98bd106700 1 -- 172.21.15.67:0/4061868281 >> 172.21.15.67:0/4061868281 conn(0x7f98b8103380 msgr2=0x7f98b8103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:40.533 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.912+0000 7f98bd106700 1 -- 172.21.15.67:0/4061868281 shutdown_connections 2024-05-03T17:12:40.533 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:39.912+0000 7f98bd106700 1 -- 172.21.15.67:0/4061868281 wait complete. 2024-05-03T17:12:40.533 INFO:teuthology.orchestra.run.smithi067.stdout:mgr epoch 5 is available 2024-05-03T17:12:40.533 INFO:teuthology.orchestra.run.smithi067.stdout:Setting orchestrator backend to cephadm... 2024-05-03T17:12:41.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:40 smithi067 ceph-mon[29731]: [03/May/2024:17:12:39] ENGINE Bus STARTING 2024-05-03T17:12:41.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:40 smithi067 ceph-mon[29731]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-03T17:12:41.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:40 smithi067 ceph-mon[29731]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-03T17:12:41.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:40 smithi067 ceph-mon[29731]: [03/May/2024:17:12:39] ENGINE Serving on https://172.21.15.67:7150 2024-05-03T17:12:41.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:40 smithi067 ceph-mon[29731]: [03/May/2024:17:12:40] ENGINE Serving on http://172.21.15.67:8765 2024-05-03T17:12:41.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:40 smithi067 ceph-mon[29731]: [03/May/2024:17:12:40] ENGINE Bus STARTED 2024-05-03T17:12:41.182 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:40 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:12:41.704 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.935+0000 7f50efcdb700 1 Processor -- start 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.936+0000 7f50efcdb700 1 -- start start 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.936+0000 7f50efcdb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 0x7f50e810a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.936+0000 7f50efcdb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50e810aea0 con 0x7f50e810a540 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.936+0000 7f50eda77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 0x7f50e810a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.936+0000 7f50eda77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 0x7f50e810a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40756/0 (socket says 172.21.15.67:40756) 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.936+0000 7f50eda77700 1 -- 172.21.15.67:0/442264457 learned_addr learned my addr 172.21.15.67:0/442264457 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.936+0000 7f50eda77700 1 -- 172.21.15.67:0/442264457 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e8108bc0 con 0x7f50e810a540 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.937+0000 7f50eda77700 1 --2- 172.21.15.67:0/442264457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 0x7f50e810a960 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f50d80098a0 tx=0x7f50d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4db29ebddcde7338 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.937+0000 7f50eca75700 1 -- 172.21.15.67:0/442264457 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50d8004160 con 0x7f50e810a540 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.937+0000 7f50eca75700 1 -- 172.21.15.67:0/442264457 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f50d8004300 con 0x7f50e810a540 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.937+0000 7f50eca75700 1 -- 172.21.15.67:0/442264457 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50d8011af0 con 0x7f50e810a540 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.937+0000 7f50efcdb700 1 -- 172.21.15.67:0/442264457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 msgr2=0x7f50e810a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:41.705 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.937+0000 7f50efcdb700 1 --2- 172.21.15.67:0/442264457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 0x7f50e810a960 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f50d80098a0 tx=0x7f50d8009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.937+0000 7f50efcdb700 1 -- 172.21.15.67:0/442264457 shutdown_connections 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.937+0000 7f50efcdb700 1 --2- 172.21.15.67:0/442264457 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 0x7f50e810a960 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.937+0000 7f50efcdb700 1 -- 172.21.15.67:0/442264457 >> 172.21.15.67:0/442264457 conn(0x7f50e8104580 msgr2=0x7f50e81069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.938+0000 7f50efcdb700 1 -- 172.21.15.67:0/442264457 shutdown_connections 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.938+0000 7f50efcdb700 1 -- 172.21.15.67:0/442264457 wait complete. 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.938+0000 7f50efcdb700 1 Processor -- start 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.938+0000 7f50efcdb700 1 -- start start 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.938+0000 7f50efcdb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 0x7f50e819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.938+0000 7f50efcdb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f50e810d320 con 0x7f50e810a540 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.938+0000 7f50eda77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 0x7f50e819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.938+0000 7f50eda77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 0x7f50e819ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40758/0 (socket says 172.21.15.67:40758) 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.938+0000 7f50eda77700 1 -- 172.21.15.67:0/4232353550 learned_addr learned my addr 172.21.15.67:0/4232353550 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:41.706 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.939+0000 7f50eda77700 1 -- 172.21.15.67:0/4232353550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50e819d410 con 0x7f50e810a540 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.939+0000 7f50eda77700 1 --2- 172.21.15.67:0/4232353550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 0x7f50e819ced0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f50d8009fb0 tx=0x7f50d8011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.939+0000 7f50deffd700 1 -- 172.21.15.67:0/4232353550 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50d80036a0 con 0x7f50e810a540 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.939+0000 7f50efcdb700 1 -- 172.21.15.67:0/4232353550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50e819d6a0 con 0x7f50e810a540 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.939+0000 7f50deffd700 1 -- 172.21.15.67:0/4232353550 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f50d801a430 con 0x7f50e810a540 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.939+0000 7f50deffd700 1 -- 172.21.15.67:0/4232353550 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f50d8019550 con 0x7f50e810a540 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.939+0000 7f50efcdb700 1 -- 172.21.15.67:0/4232353550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50e81a0390 con 0x7f50e810a540 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.940+0000 7f50deffd700 1 -- 172.21.15.67:0/4232353550 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7f50d8019850 con 0x7f50e810a540 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.940+0000 7f50efcdb700 1 -- 172.21.15.67:0/4232353550 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50e8064b00 con 0x7f50e810a540 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.940+0000 7f50deffd700 1 --2- 172.21.15.67:0/4232353550 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f50d403e2f0 0x7f50d40407b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.940+0000 7f50deffd700 1 -- 172.21.15.67:0/4232353550 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f50d80537e0 con 0x7f50e810a540 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.940+0000 7f50ed276700 1 --2- 172.21.15.67:0/4232353550 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f50d403e2f0 0x7f50d40407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.941+0000 7f50ed276700 1 --2- 172.21.15.67:0/4232353550 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f50d403e2f0 0x7f50d40407b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f50e4006fd0 tx=0x7f50e4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.946+0000 7f50deffd700 1 -- 172.21.15.67:0/4232353550 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f50d8023390 con 0x7f50e810a540 2024-05-03T17:12:41.707 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:40.957+0000 7f50deffd700 1 -- 172.21.15.67:0/4232353550 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f50d802d440 con 0x7f50e810a540 2024-05-03T17:12:41.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:41.179+0000 7f50efcdb700 1 -- 172.21.15.67:0/4232353550 --> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f50e81057d0 con 0x7f50d403e2f0 2024-05-03T17:12:41.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:41.184+0000 7f50deffd700 1 -- 172.21.15.67:0/4232353550 <== mgr.14120 v2:172.21.15.67:6800/3335149382 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f50e81057d0 con 0x7f50d403e2f0 2024-05-03T17:12:41.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:41.187+0000 7f50efcdb700 1 -- 172.21.15.67:0/4232353550 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f50d403e2f0 msgr2=0x7f50d40407b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:41.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:41.187+0000 7f50efcdb700 1 --2- 172.21.15.67:0/4232353550 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f50d403e2f0 0x7f50d40407b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f50e4006fd0 tx=0x7f50e4006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:41.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:41.187+0000 7f50efcdb700 1 -- 172.21.15.67:0/4232353550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 msgr2=0x7f50e819ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:41.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:41.187+0000 7f50efcdb700 1 --2- 172.21.15.67:0/4232353550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 0x7f50e819ced0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f50d8009fb0 tx=0x7f50d8011940 comp rx=0 tx=0).stop 2024-05-03T17:12:41.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:41.187+0000 7f50efcdb700 1 -- 172.21.15.67:0/4232353550 shutdown_connections 2024-05-03T17:12:41.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:41.187+0000 7f50efcdb700 1 --2- 172.21.15.67:0/4232353550 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f50d403e2f0 0x7f50d40407b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:41.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:41.187+0000 7f50efcdb700 1 --2- 172.21.15.67:0/4232353550 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f50e810a540 0x7f50e819ced0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:41.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:41.187+0000 7f50efcdb700 1 -- 172.21.15.67:0/4232353550 >> 172.21.15.67:0/4232353550 conn(0x7f50e8104580 msgr2=0x7f50e81050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:41.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:41.187+0000 7f50efcdb700 1 -- 172.21.15.67:0/4232353550 shutdown_connections 2024-05-03T17:12:41.708 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:41.187+0000 7f50efcdb700 1 -- 172.21.15.67:0/4232353550 wait complete. 2024-05-03T17:12:42.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:41 smithi067 ceph-mon[29731]: mgrmap e8: x(active, since 2s) 2024-05-03T17:12:42.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:41 smithi067 ceph-mon[29731]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:12:42.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:41 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:42.083 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:41 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:12:42.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-03T17:12:42.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.106+0000 7fe6fd6bb700 1 Processor -- start 2024-05-03T17:12:42.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.106+0000 7fe6fd6bb700 1 -- start start 2024-05-03T17:12:42.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.106+0000 7fe6fd6bb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 0x7fe6f81097c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:42.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.106+0000 7fe6fd6bb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f8109d90 con 0x7fe6f81093a0 2024-05-03T17:12:42.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.107+0000 7fe6f6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 0x7fe6f81097c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:42.920 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.107+0000 7fe6f6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 0x7fe6f81097c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40760/0 (socket says 172.21.15.67:40760) 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.107+0000 7fe6f6ffd700 1 -- 172.21.15.67:0/1908096262 learned_addr learned my addr 172.21.15.67:0/1908096262 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.107+0000 7fe6f6ffd700 1 -- 172.21.15.67:0/1908096262 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f8107fb0 con 0x7fe6f81093a0 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.107+0000 7fe6f6ffd700 1 --2- 172.21.15.67:0/1908096262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 0x7fe6f81097c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009e80 tx=0x7fe6e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70879fcb6dfb0eff server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.107+0000 7fe6f5ffb700 1 -- 172.21.15.67:0/1908096262 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6e8017070 con 0x7fe6f81093a0 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.107+0000 7fe6f5ffb700 1 -- 172.21.15.67:0/1908096262 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe6e800b8c0 con 0x7fe6f81093a0 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.107+0000 7fe6f5ffb700 1 -- 172.21.15.67:0/1908096262 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6e8004490 con 0x7fe6f81093a0 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.108+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/1908096262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 msgr2=0x7fe6f81097c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.108+0000 7fe6fd6bb700 1 --2- 172.21.15.67:0/1908096262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 0x7fe6f81097c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8009e80 tx=0x7fe6e800b330 comp rx=0 tx=0).stop 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.108+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/1908096262 shutdown_connections 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.108+0000 7fe6fd6bb700 1 --2- 172.21.15.67:0/1908096262 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 0x7fe6f81097c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.108+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/1908096262 >> 172.21.15.67:0/1908096262 conn(0x7fe6f8102b00 msgr2=0x7fe6f8104f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.108+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/1908096262 shutdown_connections 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.108+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/1908096262 wait complete. 2024-05-03T17:12:42.921 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.109+0000 7fe6fd6bb700 1 Processor -- start 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.109+0000 7fe6fd6bb700 1 -- start start 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.109+0000 7fe6fd6bb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 0x7fe6f8192eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.109+0000 7fe6fd6bb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f810c1f0 con 0x7fe6f81093a0 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.109+0000 7fe6f6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 0x7fe6f8192eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.109+0000 7fe6f6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 0x7fe6f8192eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40768/0 (socket says 172.21.15.67:40768) 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.109+0000 7fe6f6ffd700 1 -- 172.21.15.67:0/679545210 learned_addr learned my addr 172.21.15.67:0/679545210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.109+0000 7fe6f6ffd700 1 -- 172.21.15.67:0/679545210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f81933f0 con 0x7fe6f81093a0 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.110+0000 7fe6f6ffd700 1 --2- 172.21.15.67:0/679545210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 0x7fe6f8192eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800b7b0 tx=0x7fe6e800be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.110+0000 7fe6effff700 1 -- 172.21.15.67:0/679545210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6e8020020 con 0x7fe6f81093a0 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.110+0000 7fe6effff700 1 -- 172.21.15.67:0/679545210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe6e8012430 con 0x7fe6f81093a0 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.110+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/679545210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6f81915e0 con 0x7fe6f81093a0 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.110+0000 7fe6effff700 1 -- 172.21.15.67:0/679545210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6e8017040 con 0x7fe6f81093a0 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.110+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/679545210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6f8191b20 con 0x7fe6f81093a0 2024-05-03T17:12:42.922 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.111+0000 7fe6effff700 1 -- 172.21.15.67:0/679545210 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fe6e8023020 con 0x7fe6f81093a0 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.111+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/679545210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6f810c370 con 0x7fe6f81093a0 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.111+0000 7fe6effff700 1 --2- 172.21.15.67:0/679545210 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7fe6e403e470 0x7fe6e4040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.111+0000 7fe6effff700 1 -- 172.21.15.67:0/679545210 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe6e800f050 con 0x7fe6f81093a0 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.111+0000 7fe6f67fc700 1 --2- 172.21.15.67:0/679545210 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7fe6e403e470 0x7fe6e4040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.112+0000 7fe6f67fc700 1 --2- 172.21.15.67:0/679545210 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7fe6e403e470 0x7fe6e4040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0006fd0 tx=0x7fe6e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.117+0000 7fe6effff700 1 -- 172.21.15.67:0/679545210 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe6e802a020 con 0x7fe6f81093a0 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.337+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/679545210 --> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fe6f8196d70 con 0x7fe6e403e470 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.337+0000 7fe6effff700 1 -- 172.21.15.67:0/679545210 <== mgr.14120 v2:172.21.15.67:6800/3335149382 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fe6f8196d70 con 0x7fe6e403e470 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.340+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/679545210 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7fe6e403e470 msgr2=0x7fe6e4040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.340+0000 7fe6fd6bb700 1 --2- 172.21.15.67:0/679545210 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7fe6e403e470 0x7fe6e4040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0006fd0 tx=0x7fe6e0006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.340+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/679545210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 msgr2=0x7fe6f8192eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.340+0000 7fe6fd6bb700 1 --2- 172.21.15.67:0/679545210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 0x7fe6f8192eb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe6e800b7b0 tx=0x7fe6e800be20 comp rx=0 tx=0).stop 2024-05-03T17:12:42.923 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.340+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/679545210 shutdown_connections 2024-05-03T17:12:42.924 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.340+0000 7fe6fd6bb700 1 --2- 172.21.15.67:0/679545210 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7fe6e403e470 0x7fe6e4040930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:42.924 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.340+0000 7fe6fd6bb700 1 --2- 172.21.15.67:0/679545210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe6f81093a0 0x7fe6f8192eb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:42.924 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.340+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/679545210 >> 172.21.15.67:0/679545210 conn(0x7fe6f8102b00 msgr2=0x7fe6f81039a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:42.924 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.340+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/679545210 shutdown_connections 2024-05-03T17:12:42.924 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:42.340+0000 7fe6fd6bb700 1 -- 172.21.15.67:0/679545210 wait complete. 2024-05-03T17:12:42.924 INFO:teuthology.orchestra.run.smithi067.stdout:Generating ssh key... 2024-05-03T17:12:43.162 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:42 smithi067 ceph-mon[29731]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:12:43.954 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: Generating public/private rsa key pair. 2024-05-03T17:12:43.954 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: Your identification has been saved in /tmp/tmpeegqhuat/key. 2024-05-03T17:12:43.954 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: Your public key has been saved in /tmp/tmpeegqhuat/key.pub. 2024-05-03T17:12:43.954 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: The key fingerprint is: 2024-05-03T17:12:43.954 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: SHA256:HF+esqmt6r1A7qbJvXMbVCxaxLZcSWvsYgmKNxY1TrU ceph-20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:12:43.954 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: The key's randomart image is: 2024-05-03T17:12:43.954 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: +---[RSA 3072]----+ 2024-05-03T17:12:43.954 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: | +oo... | 2024-05-03T17:12:43.954 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: | + oo+o. | 2024-05-03T17:12:43.954 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: | . ooEo* . | 2024-05-03T17:12:43.955 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: | . o =oO o . | 2024-05-03T17:12:43.955 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: | . = o S + o | 2024-05-03T17:12:43.955 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: | o + o . + | 2024-05-03T17:12:43.955 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: | o . o | 2024-05-03T17:12:43.955 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: | . +oo.+ | 2024-05-03T17:12:43.955 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: | ++*=*+. | 2024-05-03T17:12:43.955 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: +----[SHA256]-----+ 2024-05-03T17:12:44.233 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:43 smithi067 ceph-mon[29731]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:12:44.233 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:43 smithi067 ceph-mon[29731]: Generating ssh key... 2024-05-03T17:12:44.233 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:43 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:44.234 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:43 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:44.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.368+0000 7fbd9ed10700 1 Processor -- start 2024-05-03T17:12:44.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.368+0000 7fbd9ed10700 1 -- start start 2024-05-03T17:12:44.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.368+0000 7fbd9ed10700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 0x7fbd9810a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.368+0000 7fbd9ed10700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9810aea0 con 0x7fbd9810a540 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.368+0000 7fbd9caac700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 0x7fbd9810a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.368+0000 7fbd9caac700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 0x7fbd9810a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40784/0 (socket says 172.21.15.67:40784) 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.368+0000 7fbd9caac700 1 -- 172.21.15.67:0/1091233921 learned_addr learned my addr 172.21.15.67:0/1091233921 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.368+0000 7fbd9caac700 1 -- 172.21.15.67:0/1091233921 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd98108bc0 con 0x7fbd9810a540 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.368+0000 7fbd9caac700 1 --2- 172.21.15.67:0/1091233921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 0x7fbd9810a960 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0098a0 tx=0x7fbd8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c58700b72fa21e server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.369+0000 7fbd977fe700 1 -- 172.21.15.67:0/1091233921 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd8c004160 con 0x7fbd9810a540 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.369+0000 7fbd977fe700 1 -- 172.21.15.67:0/1091233921 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbd8c004300 con 0x7fbd9810a540 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.369+0000 7fbd977fe700 1 -- 172.21.15.67:0/1091233921 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd8c011af0 con 0x7fbd9810a540 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.369+0000 7fbd9ed10700 1 -- 172.21.15.67:0/1091233921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 msgr2=0x7fbd9810a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.369+0000 7fbd9ed10700 1 --2- 172.21.15.67:0/1091233921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 0x7fbd9810a960 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c0098a0 tx=0x7fbd8c009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.369+0000 7fbd9ed10700 1 -- 172.21.15.67:0/1091233921 shutdown_connections 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.369+0000 7fbd9ed10700 1 --2- 172.21.15.67:0/1091233921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 0x7fbd9810a960 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:44.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.369+0000 7fbd9ed10700 1 -- 172.21.15.67:0/1091233921 >> 172.21.15.67:0/1091233921 conn(0x7fbd98104580 msgr2=0x7fbd981069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.369+0000 7fbd9ed10700 1 -- 172.21.15.67:0/1091233921 shutdown_connections 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.369+0000 7fbd9ed10700 1 -- 172.21.15.67:0/1091233921 wait complete. 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.370+0000 7fbd9ed10700 1 Processor -- start 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.370+0000 7fbd9ed10700 1 -- start start 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.370+0000 7fbd9ed10700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 0x7fbd9807f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.370+0000 7fbd9ed10700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbd9810d320 con 0x7fbd9810a540 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.370+0000 7fbd9caac700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 0x7fbd9807f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.370+0000 7fbd9caac700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 0x7fbd9807f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40786/0 (socket says 172.21.15.67:40786) 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.370+0000 7fbd9caac700 1 -- 172.21.15.67:0/2435132508 learned_addr learned my addr 172.21.15.67:0/2435132508 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.371+0000 7fbd9caac700 1 -- 172.21.15.67:0/2435132508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd9807f960 con 0x7fbd9810a540 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.371+0000 7fbd9caac700 1 --2- 172.21.15.67:0/2435132508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 0x7fbd9807f420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c011ac0 tx=0x7fbd8c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.371+0000 7fbd95ffb700 1 -- 172.21.15.67:0/2435132508 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd8c003970 con 0x7fbd9810a540 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.371+0000 7fbd95ffb700 1 -- 172.21.15.67:0/2435132508 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fbd8c003b10 con 0x7fbd9810a540 2024-05-03T17:12:44.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.371+0000 7fbd9ed10700 1 -- 172.21.15.67:0/2435132508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd9807db50 con 0x7fbd9810a540 2024-05-03T17:12:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.371+0000 7fbd95ffb700 1 -- 172.21.15.67:0/2435132508 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbd8c024610 con 0x7fbd9810a540 2024-05-03T17:12:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.371+0000 7fbd9ed10700 1 -- 172.21.15.67:0/2435132508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd9807e090 con 0x7fbd9810a540 2024-05-03T17:12:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.372+0000 7fbd95ffb700 1 -- 172.21.15.67:0/2435132508 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fbd8c01a430 con 0x7fbd9810a540 2024-05-03T17:12:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.372+0000 7fbd9ed10700 1 -- 172.21.15.67:0/2435132508 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd98064b00 con 0x7fbd9810a540 2024-05-03T17:12:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.372+0000 7fbd95ffb700 1 --2- 172.21.15.67:0/2435132508 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7fbd8803e470 0x7fbd88040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.372+0000 7fbd95ffb700 1 -- 172.21.15.67:0/2435132508 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbd8c053870 con 0x7fbd9810a540 2024-05-03T17:12:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.372+0000 7fbd97fff700 1 --2- 172.21.15.67:0/2435132508 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7fbd8803e470 0x7fbd88040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.373+0000 7fbd97fff700 1 --2- 172.21.15.67:0/2435132508 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7fbd8803e470 0x7fbd88040930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbd84006fd0 tx=0x7fbd84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.377+0000 7fbd95ffb700 1 -- 172.21.15.67:0/2435132508 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fbd8c00b3e0 con 0x7fbd9810a540 2024-05-03T17:12:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.598+0000 7fbd9ed10700 1 -- 172.21.15.67:0/2435132508 --> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fbd9807e7e0 con 0x7fbd8803e470 2024-05-03T17:12:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.767+0000 7fbd95ffb700 1 -- 172.21.15.67:0/2435132508 <== mgr.14120 v2:172.21.15.67:6800/3335149382 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fbd9807e7e0 con 0x7fbd8803e470 2024-05-03T17:12:44.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.770+0000 7fbd9ed10700 1 -- 172.21.15.67:0/2435132508 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7fbd8803e470 msgr2=0x7fbd88040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:44.244 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.770+0000 7fbd9ed10700 1 --2- 172.21.15.67:0/2435132508 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7fbd8803e470 0x7fbd88040930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbd84006fd0 tx=0x7fbd84006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:44.244 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.770+0000 7fbd9ed10700 1 -- 172.21.15.67:0/2435132508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 msgr2=0x7fbd9807f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:44.244 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.770+0000 7fbd9ed10700 1 --2- 172.21.15.67:0/2435132508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 0x7fbd9807f420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c011ac0 tx=0x7fbd8c011ba0 comp rx=0 tx=0).stop 2024-05-03T17:12:44.244 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.770+0000 7fbd9ed10700 1 -- 172.21.15.67:0/2435132508 shutdown_connections 2024-05-03T17:12:44.244 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.770+0000 7fbd9ed10700 1 --2- 172.21.15.67:0/2435132508 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7fbd8803e470 0x7fbd88040930 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:44.244 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.770+0000 7fbd9ed10700 1 --2- 172.21.15.67:0/2435132508 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbd9810a540 0x7fbd9807f420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:44.244 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.770+0000 7fbd9ed10700 1 -- 172.21.15.67:0/2435132508 >> 172.21.15.67:0/2435132508 conn(0x7fbd98104580 msgr2=0x7fbd98105110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:44.244 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.770+0000 7fbd9ed10700 1 -- 172.21.15.67:0/2435132508 shutdown_connections 2024-05-03T17:12:44.244 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:43.770+0000 7fbd9ed10700 1 -- 172.21.15.67:0/2435132508 wait complete. 2024-05-03T17:12:45.513 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmOjo8dWltEJGOxPvkwh30otUWA1zjSWL+5hoobQWmhu5Z8xaYm8AtC9gtK/xBWW2qTOJU9BrZFPiTcqEAtUakMHGxX2+pvmQhZSwpo8ATkZFgn9RGEDeySGbIjpf/fr38vthQtE3m8XMy2j1oPcRwBb+Mu5/E7knhIiZ7eezhcsqF2TKxIZ/Oke6KYb9FsmmKvQIWXj/QStz4/SXrvk6HNIKH+MSd6vaMY3XERNi4cUvv6xpVeJyLjTbKodX0y1ERrLUxkwBX8E8k04WvLc357Wu8ENekAXyZZMf5nIyN2CJ+BLEv+8vpFSfAv+KNrizjfNx/vj+ffQ8dJeppZk3+3cGoNpmihivl5SE1/7nK8F/ClXnIMSEJ5QlsWw6txNbMXZyt5Kw3Rs3HRvYVpQs1b9fr/51ogeT7hpFZrRWptNyHcfESWGtrw7tMbIEzeZC5eDkjyKQB+jGezSO/hz1LkRs7uyibJfivDq18XWVAjw7gHCtth5+E4k7z0Di+mE= ceph-20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:12:45.513 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.620+0000 7f18bde6c700 1 Processor -- start 2024-05-03T17:12:45.513 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.620+0000 7f18bde6c700 1 -- start start 2024-05-03T17:12:45.513 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.620+0000 7f18bde6c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 0x7f18b8107720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:45.514 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.620+0000 7f18bde6c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f18b8107cf0 con 0x7f18b8107300 2024-05-03T17:12:45.514 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.620+0000 7f18b77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 0x7f18b8107720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:45.514 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.620+0000 7f18b77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 0x7f18b8107720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40792/0 (socket says 172.21.15.67:40792) 2024-05-03T17:12:45.514 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.620+0000 7f18b77fe700 1 -- 172.21.15.67:0/1076074496 learned_addr learned my addr 172.21.15.67:0/1076074496 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:45.514 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.621+0000 7f18b77fe700 1 -- 172.21.15.67:0/1076074496 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b810a240 con 0x7f18b8107300 2024-05-03T17:12:45.514 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.621+0000 7f18b77fe700 1 --2- 172.21.15.67:0/1076074496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 0x7f18b8107720 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f18a80098a0 tx=0x7f18a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7490c395c62c4e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:45.515 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.621+0000 7f18b67fc700 1 -- 172.21.15.67:0/1076074496 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18a8004160 con 0x7f18b8107300 2024-05-03T17:12:45.515 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.621+0000 7f18b67fc700 1 -- 172.21.15.67:0/1076074496 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f18a8004300 con 0x7f18b8107300 2024-05-03T17:12:45.515 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.621+0000 7f18b67fc700 1 -- 172.21.15.67:0/1076074496 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18a8011af0 con 0x7f18b8107300 2024-05-03T17:12:45.515 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.621+0000 7f18bde6c700 1 -- 172.21.15.67:0/1076074496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 msgr2=0x7f18b8107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:45.515 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.621+0000 7f18bde6c700 1 --2- 172.21.15.67:0/1076074496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 0x7f18b8107720 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f18a80098a0 tx=0x7f18a8009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:45.515 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.621+0000 7f18bde6c700 1 -- 172.21.15.67:0/1076074496 shutdown_connections 2024-05-03T17:12:45.516 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.621+0000 7f18bde6c700 1 --2- 172.21.15.67:0/1076074496 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 0x7f18b8107720 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:45.516 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.621+0000 7f18bde6c700 1 -- 172.21.15.67:0/1076074496 >> 172.21.15.67:0/1076074496 conn(0x7f18b8102b80 msgr2=0x7f18b8104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:45.516 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.622+0000 7f18bde6c700 1 -- 172.21.15.67:0/1076074496 shutdown_connections 2024-05-03T17:12:45.516 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.622+0000 7f18bde6c700 1 -- 172.21.15.67:0/1076074496 wait complete. 2024-05-03T17:12:45.516 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.622+0000 7f18bde6c700 1 Processor -- start 2024-05-03T17:12:45.516 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.622+0000 7f18bde6c700 1 -- start start 2024-05-03T17:12:45.516 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.623+0000 7f18bde6c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 0x7f18b8109ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:45.517 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.623+0000 7f18bde6c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f18b810d470 con 0x7f18b8107300 2024-05-03T17:12:45.517 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.623+0000 7f18b77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 0x7f18b8109ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:45.517 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.623+0000 7f18b77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 0x7f18b8109ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40794/0 (socket says 172.21.15.67:40794) 2024-05-03T17:12:45.517 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.623+0000 7f18b77fe700 1 -- 172.21.15.67:0/4127158980 learned_addr learned my addr 172.21.15.67:0/4127158980 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:45.517 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.623+0000 7f18b77fe700 1 -- 172.21.15.67:0/4127158980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b810a000 con 0x7f18b8107300 2024-05-03T17:12:45.517 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.623+0000 7f18b77fe700 1 --2- 172.21.15.67:0/4127158980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 0x7f18b8109ac0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f18a80117b0 tx=0x7f18a8011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:45.518 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.623+0000 7f18b4ff9700 1 -- 172.21.15.67:0/4127158980 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18a8003af0 con 0x7f18b8107300 2024-05-03T17:12:45.518 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.623+0000 7f18b4ff9700 1 -- 172.21.15.67:0/4127158980 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f18a8020070 con 0x7f18b8107300 2024-05-03T17:12:45.518 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.623+0000 7f18bde6c700 1 -- 172.21.15.67:0/4127158980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b81081f0 con 0x7f18b8107300 2024-05-03T17:12:45.518 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.623+0000 7f18b4ff9700 1 -- 172.21.15.67:0/4127158980 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f18a8019630 con 0x7f18b8107300 2024-05-03T17:12:45.518 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.623+0000 7f18bde6c700 1 -- 172.21.15.67:0/4127158980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b8108730 con 0x7f18b8107300 2024-05-03T17:12:45.518 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.624+0000 7f18b4ff9700 1 -- 172.21.15.67:0/4127158980 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f18a80198f0 con 0x7f18b8107300 2024-05-03T17:12:45.518 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.624+0000 7f18bde6c700 1 -- 172.21.15.67:0/4127158980 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b810d5f0 con 0x7f18b8107300 2024-05-03T17:12:45.518 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.624+0000 7f18b4ff9700 1 --2- 172.21.15.67:0/4127158980 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f18a003e470 0x7f18a0040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:45.519 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.625+0000 7f18b4ff9700 1 -- 172.21.15.67:0/4127158980 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f18a8053a80 con 0x7f18b8107300 2024-05-03T17:12:45.519 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.625+0000 7f18b6ffd700 1 --2- 172.21.15.67:0/4127158980 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f18a003e470 0x7f18a0040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:45.519 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.625+0000 7f18b6ffd700 1 --2- 172.21.15.67:0/4127158980 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f18a003e470 0x7f18a0040930 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f18ac006fd0 tx=0x7f18ac006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:45.519 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.630+0000 7f18b4ff9700 1 -- 172.21.15.67:0/4127158980 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f18a8016260 con 0x7f18b8107300 2024-05-03T17:12:45.519 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.851+0000 7f18bde6c700 1 -- 172.21.15.67:0/4127158980 --> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f18b8108e70 con 0x7f18a003e470 2024-05-03T17:12:45.519 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.852+0000 7f18b4ff9700 1 -- 172.21.15.67:0/4127158980 <== mgr.14120 v2:172.21.15.67:6800/3335149382 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f18b8108e70 con 0x7f18a003e470 2024-05-03T17:12:45.519 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.854+0000 7f18bde6c700 1 -- 172.21.15.67:0/4127158980 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f18a003e470 msgr2=0x7f18a0040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:45.520 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.854+0000 7f18bde6c700 1 --2- 172.21.15.67:0/4127158980 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f18a003e470 0x7f18a0040930 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f18ac006fd0 tx=0x7f18ac006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:45.520 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.854+0000 7f18bde6c700 1 -- 172.21.15.67:0/4127158980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 msgr2=0x7f18b8109ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:45.520 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.854+0000 7f18bde6c700 1 --2- 172.21.15.67:0/4127158980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 0x7f18b8109ac0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f18a80117b0 tx=0x7f18a8011b70 comp rx=0 tx=0).stop 2024-05-03T17:12:45.520 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.854+0000 7f18bde6c700 1 -- 172.21.15.67:0/4127158980 shutdown_connections 2024-05-03T17:12:45.520 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.854+0000 7f18bde6c700 1 --2- 172.21.15.67:0/4127158980 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f18a003e470 0x7f18a0040930 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:45.520 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.854+0000 7f18bde6c700 1 --2- 172.21.15.67:0/4127158980 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f18b8107300 0x7f18b8109ac0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:45.520 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.854+0000 7f18bde6c700 1 -- 172.21.15.67:0/4127158980 >> 172.21.15.67:0/4127158980 conn(0x7f18b8102b80 msgr2=0x7f18b8103730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:45.521 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.854+0000 7f18bde6c700 1 -- 172.21.15.67:0/4127158980 shutdown_connections 2024-05-03T17:12:45.521 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:44.854+0000 7f18bde6c700 1 -- 172.21.15.67:0/4127158980 wait complete. 2024-05-03T17:12:45.521 INFO:teuthology.orchestra.run.smithi067.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-03T17:12:45.521 INFO:teuthology.orchestra.run.smithi067.stdout:Adding key to root@localhost authorized_keys... 2024-05-03T17:12:45.521 INFO:teuthology.orchestra.run.smithi067.stdout:Adding host smithi067... 2024-05-03T17:12:45.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:45 smithi067 ceph-mon[29731]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:12:46.836 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:46 smithi067 ceph-mon[29731]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi067", "addr": "172.21.15.67", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:12:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:47 smithi067 ceph-mon[29731]: Deploying cephadm binary to smithi067 2024-05-03T17:12:51.114 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Added host 'smithi067' with addr '172.21.15.67' 2024-05-03T17:12:51.114 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.988+0000 7f7015777700 1 Processor -- start 2024-05-03T17:12:51.114 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.988+0000 7f7015777700 1 -- start start 2024-05-03T17:12:51.114 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.988+0000 7f7015777700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 0x7f7010109870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:51.114 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.988+0000 7f7015777700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7010109db0 con 0x7f7010109450 2024-05-03T17:12:51.114 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.988+0000 7f700effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 0x7f7010109870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:51.114 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.988+0000 7f700effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 0x7f7010109870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40806/0 (socket says 172.21.15.67:40806) 2024-05-03T17:12:51.114 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.988+0000 7f700effd700 1 -- 172.21.15.67:0/963290341 learned_addr learned my addr 172.21.15.67:0/963290341 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:51.114 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.988+0000 7f700effd700 1 -- 172.21.15.67:0/963290341 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7010107a30 con 0x7f7010109450 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.989+0000 7f700effd700 1 --2- 172.21.15.67:0/963290341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 0x7f7010109870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f70000098a0 tx=0x7f7000009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a8d25c54738dfc server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.989+0000 7f700dffb700 1 -- 172.21.15.67:0/963290341 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7000004160 con 0x7f7010109450 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.989+0000 7f700dffb700 1 -- 172.21.15.67:0/963290341 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7000004300 con 0x7f7010109450 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.989+0000 7f700dffb700 1 -- 172.21.15.67:0/963290341 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7000011af0 con 0x7f7010109450 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.989+0000 7f7015777700 1 -- 172.21.15.67:0/963290341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 msgr2=0x7f7010109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.989+0000 7f7015777700 1 --2- 172.21.15.67:0/963290341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 0x7f7010109870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f70000098a0 tx=0x7f7000009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.989+0000 7f7015777700 1 -- 172.21.15.67:0/963290341 shutdown_connections 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.989+0000 7f7015777700 1 --2- 172.21.15.67:0/963290341 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 0x7f7010109870 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.989+0000 7f7015777700 1 -- 172.21.15.67:0/963290341 >> 172.21.15.67:0/963290341 conn(0x7f70101033f0 msgr2=0x7f7010105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.990+0000 7f7015777700 1 -- 172.21.15.67:0/963290341 shutdown_connections 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.990+0000 7f7015777700 1 -- 172.21.15.67:0/963290341 wait complete. 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.990+0000 7f7015777700 1 Processor -- start 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.990+0000 7f7015777700 1 -- start start 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.990+0000 7f7015777700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 0x7f701007d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:51.115 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.990+0000 7f7015777700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f701010c230 con 0x7f7010109450 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.991+0000 7f700effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 0x7f701007d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.991+0000 7f700effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 0x7f701007d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:40812/0 (socket says 172.21.15.67:40812) 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.991+0000 7f700effd700 1 -- 172.21.15.67:0/2029450421 learned_addr learned my addr 172.21.15.67:0/2029450421 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.991+0000 7f700effd700 1 -- 172.21.15.67:0/2029450421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f701019fc50 con 0x7f7010109450 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.991+0000 7f700effd700 1 --2- 172.21.15.67:0/2029450421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 0x7f701007d300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7000011ac0 tx=0x7f7000011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.991+0000 7f6ff7fff700 1 -- 172.21.15.67:0/2029450421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7000003970 con 0x7f7010109450 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.991+0000 7f6ff7fff700 1 -- 172.21.15.67:0/2029450421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7000003b10 con 0x7f7010109450 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.991+0000 7f7015777700 1 -- 172.21.15.67:0/2029450421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f701019fe20 con 0x7f7010109450 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.991+0000 7f6ff7fff700 1 -- 172.21.15.67:0/2029450421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7000024610 con 0x7f7010109450 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.991+0000 7f7015777700 1 -- 172.21.15.67:0/2029450421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70101a01c0 con 0x7f7010109450 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.992+0000 7f6ff7fff700 1 -- 172.21.15.67:0/2029450421 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f700001a430 con 0x7f7010109450 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.992+0000 7f7015777700 1 -- 172.21.15.67:0/2029450421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7010064b00 con 0x7f7010109450 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.992+0000 7f6ff7fff700 1 --2- 172.21.15.67:0/2029450421 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f6ff803e470 0x7f6ff8040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:51.116 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.993+0000 7f6ff7fff700 1 -- 172.21.15.67:0/2029450421 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f70000538a0 con 0x7f7010109450 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.993+0000 7f700e7fc700 1 --2- 172.21.15.67:0/2029450421 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f6ff803e470 0x7f6ff8040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.993+0000 7f700e7fc700 1 --2- 172.21.15.67:0/2029450421 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f6ff803e470 0x7f6ff8040930 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7004006fd0 tx=0x7f7004006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:45.998+0000 7f6ff7fff700 1 -- 172.21.15.67:0/2029450421 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f700000b3e0 con 0x7f7010109450 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:46.217+0000 7f7015777700 1 -- 172.21.15.67:0/2029450421 --> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi067", "addr": "172.21.15.67", "target": ["mon-mgr", ""]}) -- 0x7f7010104660 con 0x7f6ff803e470 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:50.498+0000 7f6ff7fff700 1 -- 172.21.15.67:0/2029450421 <== mgr.14120 v2:172.21.15.67:6800/3335149382 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f7010104660 con 0x7f6ff803e470 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:50.501+0000 7f7015777700 1 -- 172.21.15.67:0/2029450421 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f6ff803e470 msgr2=0x7f6ff8040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:50.501+0000 7f7015777700 1 --2- 172.21.15.67:0/2029450421 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f6ff803e470 0x7f6ff8040930 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7004006fd0 tx=0x7f7004006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:50.501+0000 7f7015777700 1 -- 172.21.15.67:0/2029450421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 msgr2=0x7f701007d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:50.501+0000 7f7015777700 1 --2- 172.21.15.67:0/2029450421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 0x7f701007d300 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7000011ac0 tx=0x7f7000011ba0 comp rx=0 tx=0).stop 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:50.501+0000 7f7015777700 1 -- 172.21.15.67:0/2029450421 shutdown_connections 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:50.501+0000 7f7015777700 1 --2- 172.21.15.67:0/2029450421 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f6ff803e470 0x7f6ff8040930 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:50.501+0000 7f7015777700 1 --2- 172.21.15.67:0/2029450421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7010109450 0x7f701007d300 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:50.501+0000 7f7015777700 1 -- 172.21.15.67:0/2029450421 >> 172.21.15.67:0/2029450421 conn(0x7f70101033f0 msgr2=0x7f7010103f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:51.117 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:50.501+0000 7f7015777700 1 -- 172.21.15.67:0/2029450421 shutdown_connections 2024-05-03T17:12:51.118 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:50.501+0000 7f7015777700 1 -- 172.21.15.67:0/2029450421 wait complete. 2024-05-03T17:12:51.118 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying unmanaged mon service... 2024-05-03T17:12:51.665 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:51 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:51.666 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:51 smithi067 ceph-mon[29731]: Added host smithi067 2024-05-03T17:12:51.666 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:51 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:12:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-03T17:12:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.664+0000 7f9da1b11700 1 Processor -- start 2024-05-03T17:12:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.664+0000 7f9da1b11700 1 -- start start 2024-05-03T17:12:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.664+0000 7f9da1b11700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 0x7f9d9c10bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.665+0000 7f9da1b11700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9c07ce70 con 0x7f9d9c109990 2024-05-03T17:12:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.665+0000 7f9d9b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 0x7f9d9c10bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.665+0000 7f9d9b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 0x7f9d9c10bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36334/0 (socket says 172.21.15.67:36334) 2024-05-03T17:12:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.665+0000 7f9d9b7fe700 1 -- 172.21.15.67:0/451973451 learned_addr learned my addr 172.21.15.67:0/451973451 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.665+0000 7f9d9b7fe700 1 -- 172.21.15.67:0/451973451 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d9c10c2c0 con 0x7f9d9c109990 2024-05-03T17:12:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.665+0000 7f9d9b7fe700 1 --2- 172.21.15.67:0/451973451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 0x7f9d9c10bd80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0098a0 tx=0x7f9d8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ab0771b63f6d3a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.665+0000 7f9d9a7fc700 1 -- 172.21.15.67:0/451973451 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d8c004160 con 0x7f9d9c109990 2024-05-03T17:12:52.559 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.665+0000 7f9d9a7fc700 1 -- 172.21.15.67:0/451973451 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9d8c004300 con 0x7f9d9c109990 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.665+0000 7f9d9a7fc700 1 -- 172.21.15.67:0/451973451 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d8c011af0 con 0x7f9d9c109990 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.666+0000 7f9da1b11700 1 -- 172.21.15.67:0/451973451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 msgr2=0x7f9d9c10bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.666+0000 7f9da1b11700 1 --2- 172.21.15.67:0/451973451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 0x7f9d9c10bd80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c0098a0 tx=0x7f9d8c009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.666+0000 7f9da1b11700 1 -- 172.21.15.67:0/451973451 shutdown_connections 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.666+0000 7f9da1b11700 1 --2- 172.21.15.67:0/451973451 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 0x7f9d9c10bd80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.666+0000 7f9da1b11700 1 -- 172.21.15.67:0/451973451 >> 172.21.15.67:0/451973451 conn(0x7f9d9c103380 msgr2=0x7f9d9c1057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.666+0000 7f9da1b11700 1 -- 172.21.15.67:0/451973451 shutdown_connections 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.666+0000 7f9da1b11700 1 -- 172.21.15.67:0/451973451 wait complete. 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.667+0000 7f9da1b11700 1 Processor -- start 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.667+0000 7f9da1b11700 1 -- start start 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.667+0000 7f9da1b11700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 0x7f9d9c19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.667+0000 7f9da1b11700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9d9c10d1c0 con 0x7f9d9c109990 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.667+0000 7f9d9b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 0x7f9d9c19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.667+0000 7f9d9b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 0x7f9d9c19cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36344/0 (socket says 172.21.15.67:36344) 2024-05-03T17:12:52.560 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.667+0000 7f9d9b7fe700 1 -- 172.21.15.67:0/2670582484 learned_addr learned my addr 172.21.15.67:0/2670582484 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:52.561 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.667+0000 7f9d9b7fe700 1 -- 172.21.15.67:0/2670582484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d9c19d2e0 con 0x7f9d9c109990 2024-05-03T17:12:52.561 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.668+0000 7f9d9b7fe700 1 --2- 172.21.15.67:0/2670582484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 0x7f9d9c19cda0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c011ac0 tx=0x7f9d8c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:52.561 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.668+0000 7f9d98ff9700 1 -- 172.21.15.67:0/2670582484 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d8c003b20 con 0x7f9d9c109990 2024-05-03T17:12:52.561 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.668+0000 7f9d98ff9700 1 -- 172.21.15.67:0/2670582484 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f9d8c01a8c0 con 0x7f9d9c109990 2024-05-03T17:12:52.561 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.668+0000 7f9da1b11700 1 -- 172.21.15.67:0/2670582484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d9c19d570 con 0x7f9d9c109990 2024-05-03T17:12:52.561 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.668+0000 7f9d98ff9700 1 -- 172.21.15.67:0/2670582484 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9d8c0248d0 con 0x7f9d9c109990 2024-05-03T17:12:52.561 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.668+0000 7f9da1b11700 1 -- 172.21.15.67:0/2670582484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d9c1a0220 con 0x7f9d9c109990 2024-05-03T17:12:52.561 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.669+0000 7f9d98ff9700 1 -- 172.21.15.67:0/2670582484 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f9d8c022020 con 0x7f9d9c109990 2024-05-03T17:12:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.669+0000 7f9da1b11700 1 -- 172.21.15.67:0/2670582484 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d9c064b00 con 0x7f9d9c109990 2024-05-03T17:12:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.669+0000 7f9d98ff9700 1 --2- 172.21.15.67:0/2670582484 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f9d8803e4c0 0x7f9d88040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.669+0000 7f9d98ff9700 1 -- 172.21.15.67:0/2670582484 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9d8c053780 con 0x7f9d9c109990 2024-05-03T17:12:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.669+0000 7f9d9affd700 1 --2- 172.21.15.67:0/2670582484 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f9d8803e4c0 0x7f9d88040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.669+0000 7f9d9affd700 1 --2- 172.21.15.67:0/2670582484 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f9d8803e4c0 0x7f9d88040980 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9d84006fd0 tx=0x7f9d84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.674+0000 7f9d98ff9700 1 -- 172.21.15.67:0/2670582484 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9d8c01aa70 con 0x7f9d9c109990 2024-05-03T17:12:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.906+0000 7f9da1b11700 1 -- 172.21.15.67:0/2670582484 --> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f9d9c1045c0 con 0x7f9d8803e4c0 2024-05-03T17:12:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.913+0000 7f9d98ff9700 1 -- 172.21.15.67:0/2670582484 <== mgr.14120 v2:172.21.15.67:6800/3335149382 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f9d9c1045c0 con 0x7f9d8803e4c0 2024-05-03T17:12:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.915+0000 7f9da1b11700 1 -- 172.21.15.67:0/2670582484 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f9d8803e4c0 msgr2=0x7f9d88040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.915+0000 7f9da1b11700 1 --2- 172.21.15.67:0/2670582484 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f9d8803e4c0 0x7f9d88040980 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9d84006fd0 tx=0x7f9d84006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.915+0000 7f9da1b11700 1 -- 172.21.15.67:0/2670582484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 msgr2=0x7f9d9c19cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:52.562 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.915+0000 7f9da1b11700 1 --2- 172.21.15.67:0/2670582484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 0x7f9d9c19cda0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f9d8c011ac0 tx=0x7f9d8c011ba0 comp rx=0 tx=0).stop 2024-05-03T17:12:52.563 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.915+0000 7f9da1b11700 1 -- 172.21.15.67:0/2670582484 shutdown_connections 2024-05-03T17:12:52.563 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.915+0000 7f9da1b11700 1 --2- 172.21.15.67:0/2670582484 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f9d8803e4c0 0x7f9d88040980 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:52.563 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.915+0000 7f9da1b11700 1 --2- 172.21.15.67:0/2670582484 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d9c109990 0x7f9d9c19cda0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:52.563 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.915+0000 7f9da1b11700 1 -- 172.21.15.67:0/2670582484 >> 172.21.15.67:0/2670582484 conn(0x7f9d9c103380 msgr2=0x7f9d9c103eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:52.563 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.916+0000 7f9da1b11700 1 -- 172.21.15.67:0/2670582484 shutdown_connections 2024-05-03T17:12:52.563 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:51.916+0000 7f9da1b11700 1 -- 172.21.15.67:0/2670582484 wait complete. 2024-05-03T17:12:52.563 INFO:teuthology.orchestra.run.smithi067.stdout:Deploying unmanaged mgr service... 2024-05-03T17:12:53.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:52 smithi067 ceph-mon[29731]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:12:53.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:52 smithi067 ceph-mon[29731]: Saving service mon spec with placement count:5 2024-05-03T17:12:53.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:52 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:54.594 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:54 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:54.594 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:54 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:54.594 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:54 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:54.602 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-03T17:12:54.602 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.737+0000 7f60f68c3700 1 Processor -- start 2024-05-03T17:12:54.602 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.737+0000 7f60f68c3700 1 -- start start 2024-05-03T17:12:54.602 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.737+0000 7f60f68c3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 0x7f60f007d340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:54.602 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.737+0000 7f60f68c3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f60f007d910 con 0x7f60f007ee90 2024-05-03T17:12:54.602 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.737+0000 7f60effff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 0x7f60f007d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:54.602 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.738+0000 7f60effff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 0x7f60f007d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36348/0 (socket says 172.21.15.67:36348) 2024-05-03T17:12:54.602 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.738+0000 7f60effff700 1 -- 172.21.15.67:0/68917414 learned_addr learned my addr 172.21.15.67:0/68917414 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:54.602 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.738+0000 7f60effff700 1 -- 172.21.15.67:0/68917414 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f007da90 con 0x7f60f007ee90 2024-05-03T17:12:54.602 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.738+0000 7f60effff700 1 --2- 172.21.15.67:0/68917414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 0x7f60f007d340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f60d80098a0 tx=0x7f60d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ac811fd1f4336e server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.738+0000 7f60eeffd700 1 -- 172.21.15.67:0/68917414 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60d8004160 con 0x7f60f007ee90 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.738+0000 7f60eeffd700 1 -- 172.21.15.67:0/68917414 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60d8004300 con 0x7f60f007ee90 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.738+0000 7f60eeffd700 1 -- 172.21.15.67:0/68917414 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60d8011a20 con 0x7f60f007ee90 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.739+0000 7f60f68c3700 1 -- 172.21.15.67:0/68917414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 msgr2=0x7f60f007d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.739+0000 7f60f68c3700 1 --2- 172.21.15.67:0/68917414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 0x7f60f007d340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f60d80098a0 tx=0x7f60d8009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.739+0000 7f60f68c3700 1 -- 172.21.15.67:0/68917414 shutdown_connections 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.739+0000 7f60f68c3700 1 --2- 172.21.15.67:0/68917414 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 0x7f60f007d340 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.739+0000 7f60f68c3700 1 -- 172.21.15.67:0/68917414 >> 172.21.15.67:0/68917414 conn(0x7f60f01044e0 msgr2=0x7f60f0106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.739+0000 7f60f68c3700 1 -- 172.21.15.67:0/68917414 shutdown_connections 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.739+0000 7f60f68c3700 1 -- 172.21.15.67:0/68917414 wait complete. 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.740+0000 7f60f68c3700 1 Processor -- start 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.740+0000 7f60f68c3700 1 -- start start 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.740+0000 7f60f68c3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 0x7f60f019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:54.603 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.740+0000 7f60f68c3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f60f010f4a0 con 0x7f60f007ee90 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.740+0000 7f60effff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 0x7f60f019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.740+0000 7f60effff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 0x7f60f019ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36352/0 (socket says 172.21.15.67:36352) 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.740+0000 7f60effff700 1 -- 172.21.15.67:0/594179874 learned_addr learned my addr 172.21.15.67:0/594179874 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.740+0000 7f60effff700 1 -- 172.21.15.67:0/594179874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60f019f4c0 con 0x7f60f007ee90 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.741+0000 7f60effff700 1 --2- 172.21.15.67:0/594179874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 0x7f60f019ef80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f60d80116e0 tx=0x7f60d8011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.741+0000 7f60ed7fa700 1 -- 172.21.15.67:0/594179874 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60d8003b20 con 0x7f60f007ee90 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.741+0000 7f60f68c3700 1 -- 172.21.15.67:0/594179874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60f019f750 con 0x7f60f007ee90 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.741+0000 7f60ed7fa700 1 -- 172.21.15.67:0/594179874 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60d801b8f0 con 0x7f60f007ee90 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.741+0000 7f60ed7fa700 1 -- 172.21.15.67:0/594179874 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f60d80238c0 con 0x7f60f007ee90 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.741+0000 7f60f68c3700 1 -- 172.21.15.67:0/594179874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60f01a2440 con 0x7f60f007ee90 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.742+0000 7f60ed7fa700 1 -- 172.21.15.67:0/594179874 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f60d8023b70 con 0x7f60f007ee90 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.742+0000 7f60ed7fa700 1 --2- 172.21.15.67:0/594179874 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f60dc03e420 0x7f60dc0408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.742+0000 7f60ed7fa700 1 -- 172.21.15.67:0/594179874 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f60d8054080 con 0x7f60f007ee90 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.742+0000 7f60ef7fe700 1 --2- 172.21.15.67:0/594179874 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f60dc03e420 0x7f60dc0408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:54.604 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.742+0000 7f60f68c3700 1 -- 172.21.15.67:0/594179874 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60f0064b00 con 0x7f60f007ee90 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.743+0000 7f60ef7fe700 1 --2- 172.21.15.67:0/594179874 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f60dc03e420 0x7f60dc0408e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f60e0006fd0 tx=0x7f60e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.752+0000 7f60ed7fa700 1 -- 172.21.15.67:0/594179874 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f60d801f030 con 0x7f60f007ee90 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.990+0000 7f60f68c3700 1 -- 172.21.15.67:0/594179874 --> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f60f0105760 con 0x7f60dc03e420 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:53.998+0000 7f60ed7fa700 1 -- 172.21.15.67:0/594179874 <== mgr.14120 v2:172.21.15.67:6800/3335149382 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f60f0105760 con 0x7f60dc03e420 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:54.000+0000 7f60f68c3700 1 -- 172.21.15.67:0/594179874 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f60dc03e420 msgr2=0x7f60dc0408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:54.000+0000 7f60f68c3700 1 --2- 172.21.15.67:0/594179874 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f60dc03e420 0x7f60dc0408e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f60e0006fd0 tx=0x7f60e0006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:54.000+0000 7f60f68c3700 1 -- 172.21.15.67:0/594179874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 msgr2=0x7f60f019ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:54.000+0000 7f60f68c3700 1 --2- 172.21.15.67:0/594179874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 0x7f60f019ef80 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f60d80116e0 tx=0x7f60d8011aa0 comp rx=0 tx=0).stop 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:54.000+0000 7f60f68c3700 1 -- 172.21.15.67:0/594179874 shutdown_connections 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:54.000+0000 7f60f68c3700 1 --2- 172.21.15.67:0/594179874 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f60dc03e420 0x7f60dc0408e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:54.000+0000 7f60f68c3700 1 --2- 172.21.15.67:0/594179874 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f60f007ee90 0x7f60f019ef80 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:54.000+0000 7f60f68c3700 1 -- 172.21.15.67:0/594179874 >> 172.21.15.67:0/594179874 conn(0x7f60f01044e0 msgr2=0x7f60f0105050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:54.000+0000 7f60f68c3700 1 -- 172.21.15.67:0/594179874 shutdown_connections 2024-05-03T17:12:54.605 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:54.000+0000 7f60f68c3700 1 -- 172.21.15.67:0/594179874 wait complete. 2024-05-03T17:12:55.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:55 smithi067 ceph-mon[29731]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:12:55.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:55 smithi067 ceph-mon[29731]: Saving service mgr spec with placement count:2 2024-05-03T17:12:55.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:55 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3294477723' entity='client.admin' 2024-05-03T17:12:56.239 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.028+0000 7f13a44b2700 1 Processor -- start 2024-05-03T17:12:56.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.028+0000 7f13a44b2700 1 -- start start 2024-05-03T17:12:56.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.029+0000 7f13a44b2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 0x7f139c07dff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:56.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.029+0000 7f13a44b2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f139c07f860 con 0x7f139c07dbd0 2024-05-03T17:12:56.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.029+0000 7f13a224e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 0x7f139c07dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:56.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.029+0000 7f13a224e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 0x7f139c07dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36364/0 (socket says 172.21.15.67:36364) 2024-05-03T17:12:56.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.029+0000 7f13a224e700 1 -- 172.21.15.67:0/165156624 learned_addr learned my addr 172.21.15.67:0/165156624 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:56.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.029+0000 7f13a224e700 1 -- 172.21.15.67:0/165156624 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139c07e530 con 0x7f139c07dbd0 2024-05-03T17:12:56.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.029+0000 7f13a224e700 1 --2- 172.21.15.67:0/165156624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 0x7f139c07dff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f13980098a0 tx=0x7f1398009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8112f8d784d484bc server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:56.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.029+0000 7f13a124c700 1 -- 172.21.15.67:0/165156624 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1398004160 con 0x7f139c07dbd0 2024-05-03T17:12:56.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.029+0000 7f13a124c700 1 -- 172.21.15.67:0/165156624 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1398004300 con 0x7f139c07dbd0 2024-05-03T17:12:56.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.029+0000 7f13a124c700 1 -- 172.21.15.67:0/165156624 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1398011a20 con 0x7f139c07dbd0 2024-05-03T17:12:56.240 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.030+0000 7f13a44b2700 1 -- 172.21.15.67:0/165156624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 msgr2=0x7f139c07dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.030+0000 7f13a44b2700 1 --2- 172.21.15.67:0/165156624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 0x7f139c07dff0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f13980098a0 tx=0x7f1398009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.030+0000 7f13a44b2700 1 -- 172.21.15.67:0/165156624 shutdown_connections 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.030+0000 7f13a44b2700 1 --2- 172.21.15.67:0/165156624 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 0x7f139c07dff0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.030+0000 7f13a44b2700 1 -- 172.21.15.67:0/165156624 >> 172.21.15.67:0/165156624 conn(0x7f139c104580 msgr2=0x7f139c1069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.030+0000 7f13a44b2700 1 -- 172.21.15.67:0/165156624 shutdown_connections 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.030+0000 7f13a44b2700 1 -- 172.21.15.67:0/165156624 wait complete. 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.031+0000 7f13a44b2700 1 Processor -- start 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.031+0000 7f13a44b2700 1 -- start start 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.031+0000 7f13a44b2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 0x7f139c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.031+0000 7f13a44b2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f139c10f5e0 con 0x7f139c07dbd0 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.031+0000 7f13a224e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 0x7f139c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.031+0000 7f13a224e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 0x7f139c19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36370/0 (socket says 172.21.15.67:36370) 2024-05-03T17:12:56.241 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.031+0000 7f13a224e700 1 -- 172.21.15.67:0/3294477723 learned_addr learned my addr 172.21.15.67:0/3294477723 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.031+0000 7f13a224e700 1 -- 172.21.15.67:0/3294477723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139c19f5f0 con 0x7f139c07dbd0 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.031+0000 7f13a224e700 1 --2- 172.21.15.67:0/3294477723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 0x7f139c19f0b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f13980116e0 tx=0x7f1398011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.032+0000 7f13937fe700 1 -- 172.21.15.67:0/3294477723 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1398003b20 con 0x7f139c07dbd0 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.032+0000 7f13937fe700 1 -- 172.21.15.67:0/3294477723 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f139801b8f0 con 0x7f139c07dbd0 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.032+0000 7f13a44b2700 1 -- 172.21.15.67:0/3294477723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f139c19f880 con 0x7f139c07dbd0 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.032+0000 7f13937fe700 1 -- 172.21.15.67:0/3294477723 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f13980238c0 con 0x7f139c07dbd0 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.032+0000 7f13a44b2700 1 -- 172.21.15.67:0/3294477723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f139c1a2570 con 0x7f139c07dbd0 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.032+0000 7f13937fe700 1 -- 172.21.15.67:0/3294477723 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f1398023a60 con 0x7f139c07dbd0 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.033+0000 7f13a44b2700 1 -- 172.21.15.67:0/3294477723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f139c064b00 con 0x7f139c07dbd0 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.033+0000 7f13937fe700 1 --2- 172.21.15.67:0/3294477723 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f138803e470 0x7f1388040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.033+0000 7f13937fe700 1 -- 172.21.15.67:0/3294477723 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f13980540b0 con 0x7f139c07dbd0 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.033+0000 7f13a1a4d700 1 --2- 172.21.15.67:0/3294477723 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f138803e470 0x7f1388040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.033+0000 7f13a1a4d700 1 --2- 172.21.15.67:0/3294477723 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f138803e470 0x7f1388040930 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f138c006fd0 tx=0x7f138c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:56.242 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.038+0000 7f13937fe700 1 -- 172.21.15.67:0/3294477723 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f139801f030 con 0x7f139c07dbd0 2024-05-03T17:12:56.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.272+0000 7f13a44b2700 1 -- 172.21.15.67:0/3294477723 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f139c1a2a30 con 0x7f139c07dbd0 2024-05-03T17:12:56.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.275+0000 7f13937fe700 1 -- 172.21.15.67:0/3294477723 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f1398012040 con 0x7f139c07dbd0 2024-05-03T17:12:56.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.277+0000 7f13a44b2700 1 -- 172.21.15.67:0/3294477723 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f138803e470 msgr2=0x7f1388040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:56.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.277+0000 7f13a44b2700 1 --2- 172.21.15.67:0/3294477723 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f138803e470 0x7f1388040930 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f138c006fd0 tx=0x7f138c006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:56.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.277+0000 7f13a44b2700 1 -- 172.21.15.67:0/3294477723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 msgr2=0x7f139c19f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:56.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.277+0000 7f13a44b2700 1 --2- 172.21.15.67:0/3294477723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 0x7f139c19f0b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f13980116e0 tx=0x7f1398011aa0 comp rx=0 tx=0).stop 2024-05-03T17:12:56.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.277+0000 7f13a44b2700 1 -- 172.21.15.67:0/3294477723 shutdown_connections 2024-05-03T17:12:56.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.277+0000 7f13a44b2700 1 --2- 172.21.15.67:0/3294477723 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f138803e470 0x7f1388040930 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:56.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.277+0000 7f13a44b2700 1 --2- 172.21.15.67:0/3294477723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f139c07dbd0 0x7f139c19f0b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:56.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.277+0000 7f13a44b2700 1 -- 172.21.15.67:0/3294477723 >> 172.21.15.67:0/3294477723 conn(0x7f139c104580 msgr2=0x7f139c105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:56.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.277+0000 7f13a44b2700 1 -- 172.21.15.67:0/3294477723 shutdown_connections 2024-05-03T17:12:56.243 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:55.278+0000 7f13a44b2700 1 -- 172.21.15.67:0/3294477723 wait complete. 2024-05-03T17:12:57.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.708+0000 7f70aa5ed700 1 Processor -- start 2024-05-03T17:12:57.630 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.708+0000 7f70aa5ed700 1 -- start start 2024-05-03T17:12:57.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.708+0000 7f70aa5ed700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 0x7f70a41092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:57.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.708+0000 7f70aa5ed700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f70a4109810 con 0x7f70a4108eb0 2024-05-03T17:12:57.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.708+0000 7f70a3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 0x7f70a41092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:57.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.708+0000 7f70a3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 0x7f70a41092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38326/0 (socket says 172.21.15.67:38326) 2024-05-03T17:12:57.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.708+0000 7f70a3fff700 1 -- 172.21.15.67:0/1217189302 learned_addr learned my addr 172.21.15.67:0/1217189302 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:57.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.709+0000 7f70a3fff700 1 -- 172.21.15.67:0/1217189302 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a410a050 con 0x7f70a4108eb0 2024-05-03T17:12:57.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.709+0000 7f70a3fff700 1 --2- 172.21.15.67:0/1217189302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 0x7f70a41092d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f70940098a0 tx=0x7f7094009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a93287a5c407a115 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:57.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.709+0000 7f70a2ffd700 1 -- 172.21.15.67:0/1217189302 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7094004160 con 0x7f70a4108eb0 2024-05-03T17:12:57.631 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.709+0000 7f70a2ffd700 1 -- 172.21.15.67:0/1217189302 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7094004300 con 0x7f70a4108eb0 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.709+0000 7f70a2ffd700 1 -- 172.21.15.67:0/1217189302 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7094011a20 con 0x7f70a4108eb0 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.710+0000 7f70aa5ed700 1 -- 172.21.15.67:0/1217189302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 msgr2=0x7f70a41092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.710+0000 7f70aa5ed700 1 --2- 172.21.15.67:0/1217189302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 0x7f70a41092d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f70940098a0 tx=0x7f7094009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.710+0000 7f70aa5ed700 1 -- 172.21.15.67:0/1217189302 shutdown_connections 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.710+0000 7f70aa5ed700 1 --2- 172.21.15.67:0/1217189302 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 0x7f70a41092d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.710+0000 7f70aa5ed700 1 -- 172.21.15.67:0/1217189302 >> 172.21.15.67:0/1217189302 conn(0x7f70a4104480 msgr2=0x7f70a41068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.710+0000 7f70aa5ed700 1 -- 172.21.15.67:0/1217189302 shutdown_connections 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.710+0000 7f70aa5ed700 1 -- 172.21.15.67:0/1217189302 wait complete. 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.711+0000 7f70aa5ed700 1 Processor -- start 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.711+0000 7f70aa5ed700 1 -- start start 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.711+0000 7f70aa5ed700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 0x7f70a419cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.711+0000 7f70aa5ed700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f70a410d310 con 0x7f70a4108eb0 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.711+0000 7f70a3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 0x7f70a419cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.711+0000 7f70a3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 0x7f70a419cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38334/0 (socket says 172.21.15.67:38334) 2024-05-03T17:12:57.632 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.711+0000 7f70a3fff700 1 -- 172.21.15.67:0/2954889763 learned_addr learned my addr 172.21.15.67:0/2954889763 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.711+0000 7f70a3fff700 1 -- 172.21.15.67:0/2954889763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70a419d300 con 0x7f70a4108eb0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.712+0000 7f70a3fff700 1 --2- 172.21.15.67:0/2954889763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 0x7f70a419cdc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70940119f0 tx=0x7f7094011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.712+0000 7f70a17fa700 1 -- 172.21.15.67:0/2954889763 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70940039e0 con 0x7f70a4108eb0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.712+0000 7f70a17fa700 1 -- 172.21.15.67:0/2954889763 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f709401b8f0 con 0x7f70a4108eb0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.712+0000 7f70a17fa700 1 -- 172.21.15.67:0/2954889763 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f70940237a0 con 0x7f70a4108eb0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.712+0000 7f70aa5ed700 1 -- 172.21.15.67:0/2954889763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70a419d590 con 0x7f70a4108eb0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.712+0000 7f70aa5ed700 1 -- 172.21.15.67:0/2954889763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70a41a0280 con 0x7f70a4108eb0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.713+0000 7f70a17fa700 1 -- 172.21.15.67:0/2954889763 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f70940193d0 con 0x7f70a4108eb0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.713+0000 7f70aa5ed700 1 -- 172.21.15.67:0/2954889763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70a4064b00 con 0x7f70a4108eb0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.713+0000 7f70a17fa700 1 --2- 172.21.15.67:0/2954889763 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f708c03e420 0x7f708c0408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.713+0000 7f70a17fa700 1 -- 172.21.15.67:0/2954889763 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7094053ef0 con 0x7f70a4108eb0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.714+0000 7f70a37fe700 1 --2- 172.21.15.67:0/2954889763 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f708c03e420 0x7f708c0408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.714+0000 7f70a37fe700 1 --2- 172.21.15.67:0/2954889763 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f708c03e420 0x7f708c0408e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7098006fd0 tx=0x7f7098006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.719+0000 7f70a17fa700 1 -- 172.21.15.67:0/2954889763 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f709401f030 con 0x7f70a4108eb0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.949+0000 7f70aa5ed700 1 -- 172.21.15.67:0/2954889763 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f70a41a08a0 con 0x7f70a4108eb0 2024-05-03T17:12:57.633 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.952+0000 7f70a17fa700 1 -- 172.21.15.67:0/2954889763 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f709400f2e0 con 0x7f70a4108eb0 2024-05-03T17:12:57.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.954+0000 7f70aa5ed700 1 -- 172.21.15.67:0/2954889763 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f708c03e420 msgr2=0x7f708c0408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:57.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.954+0000 7f70aa5ed700 1 --2- 172.21.15.67:0/2954889763 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f708c03e420 0x7f708c0408e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7098006fd0 tx=0x7f7098006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:57.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.954+0000 7f70aa5ed700 1 -- 172.21.15.67:0/2954889763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 msgr2=0x7f70a419cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:57.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.954+0000 7f70aa5ed700 1 --2- 172.21.15.67:0/2954889763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 0x7f70a419cdc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f70940119f0 tx=0x7f7094011ad0 comp rx=0 tx=0).stop 2024-05-03T17:12:57.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.954+0000 7f70aa5ed700 1 -- 172.21.15.67:0/2954889763 shutdown_connections 2024-05-03T17:12:57.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.954+0000 7f70aa5ed700 1 --2- 172.21.15.67:0/2954889763 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f708c03e420 0x7f708c0408e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:57.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.954+0000 7f70aa5ed700 1 --2- 172.21.15.67:0/2954889763 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f70a4108eb0 0x7f70a419cdc0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:57.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.955+0000 7f70aa5ed700 1 -- 172.21.15.67:0/2954889763 >> 172.21.15.67:0/2954889763 conn(0x7f70a4104480 msgr2=0x7f70a4104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:57.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.955+0000 7f70aa5ed700 1 -- 172.21.15.67:0/2954889763 shutdown_connections 2024-05-03T17:12:57.634 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:56.955+0000 7f70aa5ed700 1 -- 172.21.15.67:0/2954889763 wait complete. 2024-05-03T17:12:57.634 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling the dashboard module... 2024-05-03T17:12:57.872 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:57 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:57.872 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:57 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2954889763' entity='client.admin' 2024-05-03T17:12:59.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:58 smithi067 ceph-mon[29731]: from='mgr.14120 172.21.15.67:0/618687248' entity='mgr.x' 2024-05-03T17:12:59.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:58 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/266246441' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-03T17:12:59.204 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:58 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: ignoring --setuser ceph since I am not root 2024-05-03T17:12:59.204 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:58 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: ignoring --setgroup ceph since I am not root 2024-05-03T17:12:59.204 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:59 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:59.124+0000 7f83be5cb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T17:12:59.690 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:59 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:59.315+0000 7f83be5cb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T17:12:59.690 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:59 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:59.424+0000 7f83be5cb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T17:12:59.697 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.081+0000 7f46f787c700 1 Processor -- start 2024-05-03T17:12:59.697 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.081+0000 7f46f787c700 1 -- start start 2024-05-03T17:12:59.697 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.081+0000 7f46f787c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 0x7f46f010bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:59.697 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.081+0000 7f46f787c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f46f007d710 con 0x7f46f0109980 2024-05-03T17:12:59.697 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.082+0000 7f46f5618700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 0x7f46f010bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:59.697 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.082+0000 7f46f5618700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 0x7f46f010bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38340/0 (socket says 172.21.15.67:38340) 2024-05-03T17:12:59.697 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.082+0000 7f46f5618700 1 -- 172.21.15.67:0/3716993593 learned_addr learned my addr 172.21.15.67:0/3716993593 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.082+0000 7f46f5618700 1 -- 172.21.15.67:0/3716993593 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f010c2b0 con 0x7f46f0109980 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.082+0000 7f46f5618700 1 --2- 172.21.15.67:0/3716993593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 0x7f46f010bd70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0098a0 tx=0x7f46ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c31b22a4c89589ab server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.082+0000 7f46e7fff700 1 -- 172.21.15.67:0/3716993593 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46ec004160 con 0x7f46f0109980 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.082+0000 7f46e7fff700 1 -- 172.21.15.67:0/3716993593 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46ec004300 con 0x7f46f0109980 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.082+0000 7f46e7fff700 1 -- 172.21.15.67:0/3716993593 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46ec011a20 con 0x7f46f0109980 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.083+0000 7f46f787c700 1 -- 172.21.15.67:0/3716993593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 msgr2=0x7f46f010bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.083+0000 7f46f787c700 1 --2- 172.21.15.67:0/3716993593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 0x7f46f010bd70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46ec0098a0 tx=0x7f46ec009bb0 comp rx=0 tx=0).stop 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.083+0000 7f46f787c700 1 -- 172.21.15.67:0/3716993593 shutdown_connections 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.083+0000 7f46f787c700 1 --2- 172.21.15.67:0/3716993593 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 0x7f46f010bd70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.083+0000 7f46f787c700 1 -- 172.21.15.67:0/3716993593 >> 172.21.15.67:0/3716993593 conn(0x7f46f0103390 msgr2=0x7f46f01057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.083+0000 7f46f787c700 1 -- 172.21.15.67:0/3716993593 shutdown_connections 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.084+0000 7f46f787c700 1 -- 172.21.15.67:0/3716993593 wait complete. 2024-05-03T17:12:59.698 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.084+0000 7f46f787c700 1 Processor -- start 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.084+0000 7f46f787c700 1 -- start start 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.084+0000 7f46f787c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 0x7f46f0109200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.084+0000 7f46f787c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f46f010d2b0 con 0x7f46f0109980 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.084+0000 7f46f5618700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 0x7f46f0109200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.084+0000 7f46f5618700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 0x7f46f0109200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38350/0 (socket says 172.21.15.67:38350) 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.084+0000 7f46f5618700 1 -- 172.21.15.67:0/266246441 learned_addr learned my addr 172.21.15.67:0/266246441 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.085+0000 7f46f5618700 1 -- 172.21.15.67:0/266246441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46f0109740 con 0x7f46f0109980 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.085+0000 7f46f5618700 1 --2- 172.21.15.67:0/266246441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 0x7f46f0109200 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f46ec004660 tx=0x7f46ec011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.085+0000 7f46e67fc700 1 -- 172.21.15.67:0/266246441 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46ec003aa0 con 0x7f46f0109980 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.085+0000 7f46e67fc700 1 -- 172.21.15.67:0/266246441 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f46ec01b8f0 con 0x7f46f0109980 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.085+0000 7f46f787c700 1 -- 172.21.15.67:0/266246441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46f0107930 con 0x7f46f0109980 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.085+0000 7f46e67fc700 1 -- 172.21.15.67:0/266246441 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f46ec023900 con 0x7f46f0109980 2024-05-03T17:12:59.699 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.085+0000 7f46f787c700 1 -- 172.21.15.67:0/266246441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46f0107e70 con 0x7f46f0109980 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.086+0000 7f46e67fc700 1 -- 172.21.15.67:0/266246441 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f46ec023bc0 con 0x7f46f0109980 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.086+0000 7f46e67fc700 1 --2- 172.21.15.67:0/266246441 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f46dc03e470 0x7f46dc040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.086+0000 7f46e67fc700 1 -- 172.21.15.67:0/266246441 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f46ec053fd0 con 0x7f46f0109980 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.086+0000 7f46f4e17700 1 --2- 172.21.15.67:0/266246441 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f46dc03e470 0x7f46dc040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.087+0000 7f46f787c700 1 -- 172.21.15.67:0/266246441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46f0064b00 con 0x7f46f0109980 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.087+0000 7f46f4e17700 1 --2- 172.21.15.67:0/266246441 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f46dc03e470 0x7f46dc040930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f46e0006fd0 tx=0x7f46e0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.096+0000 7f46e67fc700 1 -- 172.21.15.67:0/266246441 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f46ec01f030 con 0x7f46f0109980 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.384+0000 7f46f787c700 1 -- 172.21.15.67:0/266246441 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f46f0108480 con 0x7f46f0109980 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.941+0000 7f46e67fc700 1 -- 172.21.15.67:0/266246441 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f46ec01bde0 con 0x7f46f0109980 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.943+0000 7f46e67fc700 1 -- 172.21.15.67:0/266246441 <== mon.0 v2:172.21.15.67:3300/0 8 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f46ec02cc90 con 0x7f46f0109980 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.944+0000 7f46f787c700 1 -- 172.21.15.67:0/266246441 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f46dc03e470 msgr2=0x7f46dc040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.944+0000 7f46f787c700 1 --2- 172.21.15.67:0/266246441 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f46dc03e470 0x7f46dc040930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f46e0006fd0 tx=0x7f46e0006e40 comp rx=0 tx=0).stop 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.944+0000 7f46f787c700 1 -- 172.21.15.67:0/266246441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 msgr2=0x7f46f0109200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:12:59.700 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.944+0000 7f46f787c700 1 --2- 172.21.15.67:0/266246441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 0x7f46f0109200 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f46ec004660 tx=0x7f46ec011aa0 comp rx=0 tx=0).stop 2024-05-03T17:12:59.701 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.944+0000 7f46f787c700 1 -- 172.21.15.67:0/266246441 shutdown_connections 2024-05-03T17:12:59.701 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.944+0000 7f46f787c700 1 --2- 172.21.15.67:0/266246441 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f46dc03e470 0x7f46dc040930 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:59.701 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.944+0000 7f46f787c700 1 --2- 172.21.15.67:0/266246441 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f46f0109980 0x7f46f0109200 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:12:59.701 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.944+0000 7f46f787c700 1 -- 172.21.15.67:0/266246441 >> 172.21.15.67:0/266246441 conn(0x7f46f0103390 msgr2=0x7f46f0103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:12:59.701 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.945+0000 7f46f787c700 1 -- 172.21.15.67:0/266246441 shutdown_connections 2024-05-03T17:12:59.701 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:12:58.945+0000 7f46f787c700 1 -- 172.21.15.67:0/266246441 wait complete. 2024-05-03T17:12:59.946 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:59 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:59.750+0000 7f83be5cb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T17:12:59.946 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:12:59 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:12:59.838+0000 7f83be5cb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T17:13:00.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:59 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/266246441' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-03T17:13:00.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:12:59 smithi067 ceph-mon[29731]: mgrmap e9: x(active, since 20s) 2024-05-03T17:13:01.106 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:00 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/235551772' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-03T17:13:01.140 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T17:13:01.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-03T17:13:01.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-03T17:13:01.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-03T17:13:01.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-03T17:13:01.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T17:13:01.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.163+0000 7f438e818700 1 Processor -- start 2024-05-03T17:13:01.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.163+0000 7f438e818700 1 -- start start 2024-05-03T17:13:01.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.163+0000 7f438e818700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 0x7f438810bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:01.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.163+0000 7f438e818700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f438807d720 con 0x7f43881099b0 2024-05-03T17:13:01.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.163+0000 7f4387fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 0x7f438810bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:01.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.163+0000 7f4387fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 0x7f438810bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38374/0 (socket says 172.21.15.67:38374) 2024-05-03T17:13:01.141 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.163+0000 7f4387fff700 1 -- 172.21.15.67:0/576406629 learned_addr learned my addr 172.21.15.67:0/576406629 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.163+0000 7f4387fff700 1 -- 172.21.15.67:0/576406629 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f438810c2e0 con 0x7f43881099b0 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.164+0000 7f4387fff700 1 --2- 172.21.15.67:0/576406629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 0x7f438810bda0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f43780098a0 tx=0x7f4378009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fce69f5b1e39354 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.164+0000 7f4386ffd700 1 -- 172.21.15.67:0/576406629 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4378004160 con 0x7f43881099b0 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.164+0000 7f4386ffd700 1 -- 172.21.15.67:0/576406629 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4378004300 con 0x7f43881099b0 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.164+0000 7f4386ffd700 1 -- 172.21.15.67:0/576406629 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4378011a20 con 0x7f43881099b0 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.164+0000 7f438e818700 1 -- 172.21.15.67:0/576406629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 msgr2=0x7f438810bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.164+0000 7f438e818700 1 --2- 172.21.15.67:0/576406629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 0x7f438810bda0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f43780098a0 tx=0x7f4378009bb0 comp rx=0 tx=0).stop 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.165+0000 7f438e818700 1 -- 172.21.15.67:0/576406629 shutdown_connections 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.165+0000 7f438e818700 1 --2- 172.21.15.67:0/576406629 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 0x7f438810bda0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.165+0000 7f438e818700 1 -- 172.21.15.67:0/576406629 >> 172.21.15.67:0/576406629 conn(0x7f43881033e0 msgr2=0x7f4388105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.165+0000 7f438e818700 1 -- 172.21.15.67:0/576406629 shutdown_connections 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.165+0000 7f438e818700 1 -- 172.21.15.67:0/576406629 wait complete. 2024-05-03T17:13:01.142 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.165+0000 7f438e818700 1 Processor -- start 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.165+0000 7f438e818700 1 -- start start 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.166+0000 7f438e818700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 0x7f4388109230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.166+0000 7f438e818700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f438810d2e0 con 0x7f43881099b0 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.166+0000 7f4387fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 0x7f4388109230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.166+0000 7f4387fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 0x7f4388109230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38378/0 (socket says 172.21.15.67:38378) 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.166+0000 7f4387fff700 1 -- 172.21.15.67:0/235551772 learned_addr learned my addr 172.21.15.67:0/235551772 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.166+0000 7f4387fff700 1 -- 172.21.15.67:0/235551772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4388109770 con 0x7f43881099b0 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.166+0000 7f4387fff700 1 --2- 172.21.15.67:0/235551772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 0x7f4388109230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f43780116e0 tx=0x7f4378011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.166+0000 7f43857fa700 1 -- 172.21.15.67:0/235551772 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4378003b20 con 0x7f43881099b0 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.166+0000 7f438e818700 1 -- 172.21.15.67:0/235551772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4388107960 con 0x7f43881099b0 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.166+0000 7f438e818700 1 -- 172.21.15.67:0/235551772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4388107ea0 con 0x7f43881099b0 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.166+0000 7f43857fa700 1 -- 172.21.15.67:0/235551772 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f437801b8f0 con 0x7f43881099b0 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.166+0000 7f43857fa700 1 -- 172.21.15.67:0/235551772 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f43780238c0 con 0x7f43881099b0 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.167+0000 7f43857fa700 1 -- 172.21.15.67:0/235551772 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f4378023b40 con 0x7f43881099b0 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.167+0000 7f438e818700 1 -- 172.21.15.67:0/235551772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4388064b00 con 0x7f43881099b0 2024-05-03T17:13:01.143 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.167+0000 7f43857fa700 1 --2- 172.21.15.67:0/235551772 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f437003e4d0 0x7f4370040990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.167+0000 7f43877fe700 1 -- 172.21.15.67:0/235551772 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f437003e4d0 msgr2=0x7f4370040990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3335149382 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.168+0000 7f43857fa700 1 -- 172.21.15.67:0/235551772 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4378054ec0 con 0x7f43881099b0 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.168+0000 7f43877fe700 1 --2- 172.21.15.67:0/235551772 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f437003e4d0 0x7f4370040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.173+0000 7f43857fa700 1 -- 172.21.15.67:0/235551772 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f437801f030 con 0x7f43881099b0 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.369+0000 7f43877fe700 1 -- 172.21.15.67:0/235551772 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f437003e4d0 msgr2=0x7f4370040990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3335149382 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.369+0000 7f43877fe700 1 --2- 172.21.15.67:0/235551772 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f437003e4d0 0x7f4370040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.496+0000 7f438e818700 1 -- 172.21.15.67:0/235551772 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f43881081e0 con 0x7f43881099b0 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.496+0000 7f43857fa700 1 -- 172.21.15.67:0/235551772 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f43881081e0 con 0x7f43881099b0 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.499+0000 7f438e818700 1 -- 172.21.15.67:0/235551772 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f437003e4d0 msgr2=0x7f4370040990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.499+0000 7f438e818700 1 --2- 172.21.15.67:0/235551772 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f437003e4d0 0x7f4370040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.499+0000 7f438e818700 1 -- 172.21.15.67:0/235551772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 msgr2=0x7f4388109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.499+0000 7f438e818700 1 --2- 172.21.15.67:0/235551772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 0x7f4388109230 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f43780116e0 tx=0x7f4378011aa0 comp rx=0 tx=0).stop 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.499+0000 7f438e818700 1 -- 172.21.15.67:0/235551772 shutdown_connections 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.499+0000 7f438e818700 1 --2- 172.21.15.67:0/235551772 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f437003e4d0 0x7f4370040990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:01.144 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.499+0000 7f438e818700 1 --2- 172.21.15.67:0/235551772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f43881099b0 0x7f4388109230 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:01.145 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.499+0000 7f438e818700 1 -- 172.21.15.67:0/235551772 >> 172.21.15.67:0/235551772 conn(0x7f43881033e0 msgr2=0x7f4388103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:01.145 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.499+0000 7f438e818700 1 -- 172.21.15.67:0/235551772 shutdown_connections 2024-05-03T17:13:01.145 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:00.499+0000 7f438e818700 1 -- 172.21.15.67:0/235551772 wait complete. 2024-05-03T17:13:01.145 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for the mgr to restart... 2024-05-03T17:13:01.145 INFO:teuthology.orchestra.run.smithi067.stdout:Waiting for mgr epoch 9... 2024-05-03T17:13:01.382 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:01 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:01.105+0000 7f83be5cb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T17:13:01.383 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:01 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:01.287+0000 7f83be5cb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T17:13:01.703 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:01 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:01.466+0000 7f83be5cb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T17:13:02.086 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:01 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:01.780+0000 7f83be5cb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T17:13:02.087 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:01 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:01.876+0000 7f83be5cb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T17:13:02.087 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:02 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:02.086+0000 7f83be5cb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T17:13:02.453 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:02 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:02.287+0000 7f83be5cb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T17:13:03.203 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:02 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:02.913+0000 7f83be5cb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T17:13:03.203 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:03 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:03.143+0000 7f83be5cb200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-03T17:13:03.953 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:03 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:03.584+0000 7f83be5cb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T17:13:03.953 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:03 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:03.685+0000 7f83be5cb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T17:13:04.441 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:04 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:04.069+0000 7f83be5cb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-03T17:13:04.441 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:04 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:04.440+0000 7f83be5cb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T17:13:04.703 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:04 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:04.536+0000 7f83be5cb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T17:13:05.203 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:04 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:04.791+0000 7f83be5cb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T17:13:05.203 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:04 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:04.930+0000 7f83be5cb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T17:13:05.703 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:05.251+0000 7f83be5cb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T17:13:05.703 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:05.372+0000 7f83be5cb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T17:13:05.703 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:05.468+0000 7f83be5cb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T17:13:06.378 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:06 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:06.095+0000 7f83be5cb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T17:13:06.378 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:13:06 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:13:06.321+0000 7f83be5cb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T17:13:06.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:06 smithi067 ceph-mon[29731]: Active manager daemon x restarted 2024-05-03T17:13:06.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:06 smithi067 ceph-mon[29731]: Activating manager daemon x 2024-05-03T17:13:06.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:06 smithi067 ceph-mon[29731]: osdmap e3: 0 total, 0 up, 0 in 2024-05-03T17:13:06.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:06 smithi067 ceph-mon[29731]: mgrmap e10: x(active, starting, since 0.00616153s) 2024-05-03T17:13:06.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:13:06.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-03T17:13:06.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-03T17:13:06.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-03T17:13:06.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-03T17:13:06.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:06 smithi067 ceph-mon[29731]: Manager daemon x is now available 2024-05-03T17:13:06.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:13:07.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-03T17:13:07.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-03T17:13:07.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:07 smithi067 ceph-mon[29731]: [03/May/2024:17:13:06] ENGINE Bus STARTING 2024-05-03T17:13:07.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:07 smithi067 ceph-mon[29731]: [03/May/2024:17:13:06] ENGINE Serving on http://172.21.15.67:8765 2024-05-03T17:13:07.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:07 smithi067 ceph-mon[29731]: [03/May/2024:17:13:06] ENGINE Serving on https://172.21.15.67:7150 2024-05-03T17:13:07.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:07 smithi067 ceph-mon[29731]: [03/May/2024:17:13:06] ENGINE Bus STARTED 2024-05-03T17:13:07.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:07 smithi067 ceph-mon[29731]: mgrmap e11: x(active, since 1.00602s) 2024-05-03T17:13:08.327 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout { 2024-05-03T17:13:08.327 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-03T17:13:08.327 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-03T17:13:08.327 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout } 2024-05-03T17:13:08.327 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.584+0000 7f460f5d8700 1 Processor -- start 2024-05-03T17:13:08.327 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.584+0000 7f460f5d8700 1 -- start start 2024-05-03T17:13:08.327 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.584+0000 7f460f5d8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 0x7f46081092d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:08.327 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.584+0000 7f460f5d8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4608109810 con 0x7f4608108eb0 2024-05-03T17:13:08.327 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.584+0000 7f460d374700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 0x7f46081092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.584+0000 7f460d374700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 0x7f46081092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38380/0 (socket says 172.21.15.67:38380) 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.584+0000 7f460d374700 1 -- 172.21.15.67:0/888240891 learned_addr learned my addr 172.21.15.67:0/888240891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.584+0000 7f460d374700 1 -- 172.21.15.67:0/888240891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f460810a050 con 0x7f4608108eb0 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.585+0000 7f460d374700 1 --2- 172.21.15.67:0/888240891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 0x7f46081092d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f45f80098a0 tx=0x7f45f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134ff292fe3dacf4 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.585+0000 7f45fffff700 1 -- 172.21.15.67:0/888240891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45f8004160 con 0x7f4608108eb0 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.585+0000 7f45fffff700 1 -- 172.21.15.67:0/888240891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45f8004300 con 0x7f4608108eb0 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.585+0000 7f45fffff700 1 -- 172.21.15.67:0/888240891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45f8011a20 con 0x7f4608108eb0 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.586+0000 7f460f5d8700 1 -- 172.21.15.67:0/888240891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 msgr2=0x7f46081092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.586+0000 7f460f5d8700 1 --2- 172.21.15.67:0/888240891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 0x7f46081092d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f45f80098a0 tx=0x7f45f8009bb0 comp rx=0 tx=0).stop 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.586+0000 7f460f5d8700 1 -- 172.21.15.67:0/888240891 shutdown_connections 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.586+0000 7f460f5d8700 1 --2- 172.21.15.67:0/888240891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 0x7f46081092d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.586+0000 7f460f5d8700 1 -- 172.21.15.67:0/888240891 >> 172.21.15.67:0/888240891 conn(0x7f4608104480 msgr2=0x7f46081068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.586+0000 7f460f5d8700 1 -- 172.21.15.67:0/888240891 shutdown_connections 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.586+0000 7f460f5d8700 1 -- 172.21.15.67:0/888240891 wait complete. 2024-05-03T17:13:08.328 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.587+0000 7f460f5d8700 1 Processor -- start 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.587+0000 7f460f5d8700 1 -- start start 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.587+0000 7f460f5d8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 0x7f460819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.587+0000 7f460f5d8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f460810d310 con 0x7f4608108eb0 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.587+0000 7f460d374700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 0x7f460819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.587+0000 7f460d374700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 0x7f460819cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:38390/0 (socket says 172.21.15.67:38390) 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.587+0000 7f460d374700 1 -- 172.21.15.67:0/1846167183 learned_addr learned my addr 172.21.15.67:0/1846167183 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.587+0000 7f460d374700 1 -- 172.21.15.67:0/1846167183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f460819d300 con 0x7f4608108eb0 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.587+0000 7f460d374700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 0x7f460819cdc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f45f80119f0 tx=0x7f45f8011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.588+0000 7f45fe7fc700 1 -- 172.21.15.67:0/1846167183 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45f80039e0 con 0x7f4608108eb0 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.588+0000 7f45fe7fc700 1 -- 172.21.15.67:0/1846167183 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45f801b8f0 con 0x7f4608108eb0 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.588+0000 7f460f5d8700 1 -- 172.21.15.67:0/1846167183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f460819d590 con 0x7f4608108eb0 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.588+0000 7f45fe7fc700 1 -- 172.21.15.67:0/1846167183 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45f80237a0 con 0x7f4608108eb0 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.588+0000 7f460f5d8700 1 -- 172.21.15.67:0/1846167183 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46081a0280 con 0x7f4608108eb0 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.589+0000 7f45fe7fc700 1 -- 172.21.15.67:0/1846167183 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f45f802c3f0 con 0x7f4608108eb0 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.589+0000 7f45fe7fc700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 0x7f45f4040990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:08.329 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.589+0000 7f45fe7fc700 1 -- 172.21.15.67:0/1846167183 --> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f45f40410a0 con 0x7f45f403e4d0 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.589+0000 7f460cb73700 1 -- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 msgr2=0x7f45f4040990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3335149382 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.589+0000 7f45fe7fc700 1 -- 172.21.15.67:0/1846167183 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f45f8053ec0 con 0x7f4608108eb0 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.589+0000 7f460cb73700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 0x7f45f4040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.790+0000 7f460cb73700 1 -- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 msgr2=0x7f45f4040990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3335149382 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:01.790+0000 7f460cb73700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 0x7f45f4040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:02.191+0000 7f460cb73700 1 -- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 msgr2=0x7f45f4040990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3335149382 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:02.191+0000 7f460cb73700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 0x7f45f4040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:02.992+0000 7f460cb73700 1 -- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 msgr2=0x7f45f4040990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3335149382 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:02.992+0000 7f460cb73700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 0x7f45f4040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:04.593+0000 7f460cb73700 1 -- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 msgr2=0x7f45f4040990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.67:6800/3335149382 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:04.593+0000 7f460cb73700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 0x7f45f4040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:06.328+0000 7f45fe7fc700 1 -- 172.21.15.67:0/1846167183 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mgrmap(e 10) ==== 50002+0+0 (secure 0 0 0) 0x7f45f8019cb0 con 0x7f4608108eb0 2024-05-03T17:13:08.330 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:06.328+0000 7f45fe7fc700 1 -- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 msgr2=0x7f45f4040990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:06.328+0000 7f45fe7fc700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 0x7f45f4040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.327+0000 7f45fe7fc700 1 -- 172.21.15.67:0/1846167183 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 11) ==== 50129+0+0 (secure 0 0 0) 0x7f45f8055ed0 con 0x7f4608108eb0 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.327+0000 7f45fe7fc700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45f4041f50 0x7f45f4044340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.327+0000 7f45fe7fc700 1 -- 172.21.15.67:0/1846167183 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f45f8021e80 con 0x7f45f4041f50 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.328+0000 7f460cb73700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45f4041f50 0x7f45f4044340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.328+0000 7f460cb73700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45f4041f50 0x7f45f4044340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4604003a10 tx=0x7f4604009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.329+0000 7f45fe7fc700 1 -- 172.21.15.67:0/1846167183 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f45f8021e80 con 0x7f45f4041f50 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.337+0000 7f460f5d8700 1 -- 172.21.15.67:0/1846167183 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f460806a5e0 con 0x7f45f4041f50 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.338+0000 7f45fe7fc700 1 -- 172.21.15.67:0/1846167183 <== mgr.14152 v2:172.21.15.67:6800/3284262914 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f460806a5e0 con 0x7f45f4041f50 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.338+0000 7f460f5d8700 1 -- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45f4041f50 msgr2=0x7f45f4044340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.338+0000 7f460f5d8700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45f4041f50 0x7f45f4044340 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4604003a10 tx=0x7f4604009270 comp rx=0 tx=0).stop 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.338+0000 7f460f5d8700 1 -- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 msgr2=0x7f460819cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:08.331 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.338+0000 7f460f5d8700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 0x7f460819cdc0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f45f80119f0 tx=0x7f45f8011ad0 comp rx=0 tx=0).stop 2024-05-03T17:13:08.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.338+0000 7f460f5d8700 1 -- 172.21.15.67:0/1846167183 shutdown_connections 2024-05-03T17:13:08.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.338+0000 7f460f5d8700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45f4041f50 0x7f45f4044340 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:08.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.338+0000 7f460f5d8700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4608108eb0 0x7f460819cdc0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:08.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.338+0000 7f460f5d8700 1 --2- 172.21.15.67:0/1846167183 >> [v2:172.21.15.67:6800/3335149382,v1:172.21.15.67:6801/3335149382] conn(0x7f45f403e4d0 0x7f45f4040990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:08.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.338+0000 7f460f5d8700 1 -- 172.21.15.67:0/1846167183 >> 172.21.15.67:0/1846167183 conn(0x7f4608104480 msgr2=0x7f4608104ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:08.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.338+0000 7f460f5d8700 1 -- 172.21.15.67:0/1846167183 shutdown_connections 2024-05-03T17:13:08.332 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:07.338+0000 7f460f5d8700 1 -- 172.21.15.67:0/1846167183 wait complete. 2024-05-03T17:13:08.332 INFO:teuthology.orchestra.run.smithi067.stdout:mgr epoch 9 is available 2024-05-03T17:13:08.332 INFO:teuthology.orchestra.run.smithi067.stdout:Generating a dashboard self-signed certificate... 2024-05-03T17:13:09.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:08 smithi067 ceph-mon[29731]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-03T17:13:09.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:08 smithi067 ceph-mon[29731]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-03T17:13:09.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:08 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-03T17:13:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.800+0000 7fb84c27e700 1 Processor -- start 2024-05-03T17:13:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.800+0000 7fb84c27e700 1 -- start start 2024-05-03T17:13:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.800+0000 7fb84c27e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 0x7fb84410bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.800+0000 7fb84c27e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb84407d6d0 con 0x7fb8441099c0 2024-05-03T17:13:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.800+0000 7fb84a01a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 0x7fb84410bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.800+0000 7fb84a01a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 0x7fb84410bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42744/0 (socket says 172.21.15.67:42744) 2024-05-03T17:13:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.800+0000 7fb84a01a700 1 -- 172.21.15.67:0/2500615046 learned_addr learned my addr 172.21.15.67:0/2500615046 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.800+0000 7fb84a01a700 1 -- 172.21.15.67:0/2500615046 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb84410c2f0 con 0x7fb8441099c0 2024-05-03T17:13:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.801+0000 7fb84a01a700 1 --2- 172.21.15.67:0/2500615046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 0x7fb84410bdb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb8400098a0 tx=0x7fb840009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd23d7d4a3e1287 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:09.844 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.801+0000 7fb849018700 1 -- 172.21.15.67:0/2500615046 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb840004160 con 0x7fb8441099c0 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.801+0000 7fb849018700 1 -- 172.21.15.67:0/2500615046 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb840004300 con 0x7fb8441099c0 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.801+0000 7fb849018700 1 -- 172.21.15.67:0/2500615046 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb840011a20 con 0x7fb8441099c0 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.801+0000 7fb84c27e700 1 -- 172.21.15.67:0/2500615046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 msgr2=0x7fb84410bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.801+0000 7fb84c27e700 1 --2- 172.21.15.67:0/2500615046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 0x7fb84410bdb0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb8400098a0 tx=0x7fb840009bb0 comp rx=0 tx=0).stop 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.802+0000 7fb84c27e700 1 -- 172.21.15.67:0/2500615046 shutdown_connections 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.802+0000 7fb84c27e700 1 --2- 172.21.15.67:0/2500615046 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 0x7fb84410bdb0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.802+0000 7fb84c27e700 1 -- 172.21.15.67:0/2500615046 >> 172.21.15.67:0/2500615046 conn(0x7fb844103390 msgr2=0x7fb8441057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.802+0000 7fb84c27e700 1 -- 172.21.15.67:0/2500615046 shutdown_connections 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.802+0000 7fb84c27e700 1 -- 172.21.15.67:0/2500615046 wait complete. 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.802+0000 7fb84c27e700 1 Processor -- start 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.802+0000 7fb84c27e700 1 -- start start 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.803+0000 7fb84c27e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 0x7fb84419cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.803+0000 7fb84c27e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb84410d290 con 0x7fb8441099c0 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.803+0000 7fb84a01a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 0x7fb84419cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:09.845 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.803+0000 7fb84a01a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 0x7fb84419cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42760/0 (socket says 172.21.15.67:42760) 2024-05-03T17:13:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.803+0000 7fb84a01a700 1 -- 172.21.15.67:0/3211599609 learned_addr learned my addr 172.21.15.67:0/3211599609 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.803+0000 7fb84a01a700 1 -- 172.21.15.67:0/3211599609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb84419d2a0 con 0x7fb8441099c0 2024-05-03T17:13:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.803+0000 7fb84a01a700 1 --2- 172.21.15.67:0/3211599609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 0x7fb84419cd60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb840004660 tx=0x7fb840011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.803+0000 7fb83b7fe700 1 -- 172.21.15.67:0/3211599609 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb840003aa0 con 0x7fb8441099c0 2024-05-03T17:13:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.803+0000 7fb83b7fe700 1 -- 172.21.15.67:0/3211599609 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb84001b8f0 con 0x7fb8441099c0 2024-05-03T17:13:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.803+0000 7fb84c27e700 1 -- 172.21.15.67:0/3211599609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb84419d530 con 0x7fb8441099c0 2024-05-03T17:13:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.803+0000 7fb83b7fe700 1 -- 172.21.15.67:0/3211599609 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb840023900 con 0x7fb8441099c0 2024-05-03T17:13:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.803+0000 7fb84c27e700 1 -- 172.21.15.67:0/3211599609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8441a0220 con 0x7fb8441099c0 2024-05-03T17:13:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.804+0000 7fb83b7fe700 1 -- 172.21.15.67:0/3211599609 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 11) ==== 50129+0+0 (secure 0 0 0) 0x7fb840023b70 con 0x7fb8441099c0 2024-05-03T17:13:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.804+0000 7fb84c27e700 1 -- 172.21.15.67:0/3211599609 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb844064b00 con 0x7fb8441099c0 2024-05-03T17:13:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.804+0000 7fb83b7fe700 1 --2- 172.21.15.67:0/3211599609 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb83003e410 0x7fb8300408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:09.846 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.805+0000 7fb849819700 1 --2- 172.21.15.67:0/3211599609 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb83003e410 0x7fb8300408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.805+0000 7fb83b7fe700 1 -- 172.21.15.67:0/3211599609 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb840053ec0 con 0x7fb8441099c0 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.805+0000 7fb849819700 1 --2- 172.21.15.67:0/3211599609 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb83003e410 0x7fb8300408d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb834006fd0 tx=0x7fb834006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.810+0000 7fb83b7fe700 1 -- 172.21.15.67:0/3211599609 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb84001f030 con 0x7fb8441099c0 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:08.843+0000 7fb83b7fe700 1 -- 172.21.15.67:0/3211599609 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7fb84001b5e0 con 0x7fb8441099c0 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:09.038+0000 7fb84c27e700 1 -- 172.21.15.67:0/3211599609 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fb844104590 con 0x7fb83003e410 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:09.228+0000 7fb83b7fe700 1 -- 172.21.15.67:0/3211599609 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fb844104590 con 0x7fb83003e410 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:09.230+0000 7fb84c27e700 1 -- 172.21.15.67:0/3211599609 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb83003e410 msgr2=0x7fb8300408d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:09.230+0000 7fb84c27e700 1 --2- 172.21.15.67:0/3211599609 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb83003e410 0x7fb8300408d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb834006fd0 tx=0x7fb834006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:09.230+0000 7fb84c27e700 1 -- 172.21.15.67:0/3211599609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 msgr2=0x7fb84419cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:09.230+0000 7fb84c27e700 1 --2- 172.21.15.67:0/3211599609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 0x7fb84419cd60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb840004660 tx=0x7fb840011aa0 comp rx=0 tx=0).stop 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:09.230+0000 7fb84c27e700 1 -- 172.21.15.67:0/3211599609 shutdown_connections 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:09.230+0000 7fb84c27e700 1 --2- 172.21.15.67:0/3211599609 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb83003e410 0x7fb8300408d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:09.847 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:09.230+0000 7fb84c27e700 1 --2- 172.21.15.67:0/3211599609 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8441099c0 0x7fb84419cd60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:09.230+0000 7fb84c27e700 1 -- 172.21.15.67:0/3211599609 >> 172.21.15.67:0/3211599609 conn(0x7fb844103390 msgr2=0x7fb844103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:09.230+0000 7fb84c27e700 1 -- 172.21.15.67:0/3211599609 shutdown_connections 2024-05-03T17:13:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:09.230+0000 7fb84c27e700 1 -- 172.21.15.67:0/3211599609 wait complete. 2024-05-03T17:13:09.848 INFO:teuthology.orchestra.run.smithi067.stdout:Creating initial admin user... 2024-05-03T17:13:10.144 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:09 smithi067 ceph-mon[29731]: mgrmap e12: x(active, since 2s) 2024-05-03T17:13:10.144 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:10.144 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:10.144 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:11.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:10 smithi067 ceph-mon[29731]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:13:11.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:12.162 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:11 smithi067 ceph-mon[29731]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$B/55lndhIdgNa36KpDfj/OXYsd/tA3mxcOcxIgiCbcyfI.Z4U82Hi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714756390, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.250+0000 7f966e20d700 1 Processor -- start 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.250+0000 7f966e20d700 1 -- start start 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.250+0000 7f966e20d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 0x7f966810aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.250+0000 7f966e20d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f966810b040 con 0x7f966810a650 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.250+0000 7f96677fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 0x7f966810aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.250+0000 7f96677fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 0x7f966810aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42776/0 (socket says 172.21.15.67:42776) 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.250+0000 7f96677fe700 1 -- 172.21.15.67:0/2096459435 learned_addr learned my addr 172.21.15.67:0/2096459435 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.251+0000 7f96677fe700 1 -- 172.21.15.67:0/2096459435 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9668106fa0 con 0x7f966810a650 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.251+0000 7f96677fe700 1 --2- 172.21.15.67:0/2096459435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 0x7f966810aa70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f96580098a0 tx=0x7f9658009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7fad7a48ade4cf5 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.251+0000 7f96667fc700 1 -- 172.21.15.67:0/2096459435 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9658004160 con 0x7f966810a650 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.251+0000 7f96667fc700 1 -- 172.21.15.67:0/2096459435 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9658004300 con 0x7f966810a650 2024-05-03T17:13:12.171 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.251+0000 7f96667fc700 1 -- 172.21.15.67:0/2096459435 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9658011a20 con 0x7f966810a650 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.252+0000 7f966e20d700 1 -- 172.21.15.67:0/2096459435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 msgr2=0x7f966810aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.252+0000 7f966e20d700 1 --2- 172.21.15.67:0/2096459435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 0x7f966810aa70 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f96580098a0 tx=0x7f9658009bb0 comp rx=0 tx=0).stop 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.252+0000 7f966e20d700 1 -- 172.21.15.67:0/2096459435 shutdown_connections 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.252+0000 7f966e20d700 1 --2- 172.21.15.67:0/2096459435 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 0x7f966810aa70 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.252+0000 7f966e20d700 1 -- 172.21.15.67:0/2096459435 >> 172.21.15.67:0/2096459435 conn(0x7f9668102b80 msgr2=0x7f9668104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.252+0000 7f966e20d700 1 -- 172.21.15.67:0/2096459435 shutdown_connections 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.252+0000 7f966e20d700 1 -- 172.21.15.67:0/2096459435 wait complete. 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.252+0000 7f966e20d700 1 Processor -- start 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.253+0000 7f966e20d700 1 -- start start 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.253+0000 7f966e20d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 0x7f966819cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.253+0000 7f966e20d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f966810d670 con 0x7f966810a650 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.253+0000 7f96677fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 0x7f966819cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.253+0000 7f96677fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 0x7f966819cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42778/0 (socket says 172.21.15.67:42778) 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.253+0000 7f96677fe700 1 -- 172.21.15.67:0/1441364265 learned_addr learned my addr 172.21.15.67:0/1441364265 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.253+0000 7f96677fe700 1 -- 172.21.15.67:0/1441364265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f966819d430 con 0x7f966810a650 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.253+0000 7f96677fe700 1 --2- 172.21.15.67:0/1441364265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 0x7f966819cef0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f96580119f0 tx=0x7f9658011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:12.172 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.254+0000 7f9664ff9700 1 -- 172.21.15.67:0/1441364265 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96580039e0 con 0x7f966810a650 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.254+0000 7f9664ff9700 1 -- 172.21.15.67:0/1441364265 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f965801b8f0 con 0x7f966810a650 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.254+0000 7f966e20d700 1 -- 172.21.15.67:0/1441364265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f966819d6c0 con 0x7f966810a650 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.254+0000 7f9664ff9700 1 -- 172.21.15.67:0/1441364265 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f96580237a0 con 0x7f966810a650 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.254+0000 7f966e20d700 1 -- 172.21.15.67:0/1441364265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96681a03b0 con 0x7f966810a650 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.254+0000 7f9664ff9700 1 -- 172.21.15.67:0/1441364265 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f9658023aa0 con 0x7f966810a650 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.255+0000 7f9664ff9700 1 --2- 172.21.15.67:0/1441364265 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f965003e470 0x7f9650040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.255+0000 7f966e20d700 1 -- 172.21.15.67:0/1441364265 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f966810d7f0 con 0x7f966810a650 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.255+0000 7f9664ff9700 1 -- 172.21.15.67:0/1441364265 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9658053f20 con 0x7f966810a650 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.255+0000 7f9666ffd700 1 --2- 172.21.15.67:0/1441364265 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f965003e470 0x7f9650040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.255+0000 7f9666ffd700 1 --2- 172.21.15.67:0/1441364265 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f965003e470 0x7f9650040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f965c006fd0 tx=0x7f965c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.260+0000 7f9664ff9700 1 -- 172.21.15.67:0/1441364265 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f965801f030 con 0x7f966810a650 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.510+0000 7f966e20d700 1 -- 172.21.15.67:0/1441364265 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f966819dba0 con 0x7f965003e470 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.757+0000 7f9664ff9700 1 -- 172.21.15.67:0/1441364265 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f966819dba0 con 0x7f965003e470 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.759+0000 7f966e20d700 1 -- 172.21.15.67:0/1441364265 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f965003e470 msgr2=0x7f9650040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:12.173 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.759+0000 7f966e20d700 1 --2- 172.21.15.67:0/1441364265 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f965003e470 0x7f9650040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f965c006fd0 tx=0x7f965c006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:12.174 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.759+0000 7f966e20d700 1 -- 172.21.15.67:0/1441364265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 msgr2=0x7f966819cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:12.174 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.759+0000 7f966e20d700 1 --2- 172.21.15.67:0/1441364265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 0x7f966819cef0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f96580119f0 tx=0x7f9658011ad0 comp rx=0 tx=0).stop 2024-05-03T17:13:12.174 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.759+0000 7f966e20d700 1 -- 172.21.15.67:0/1441364265 shutdown_connections 2024-05-03T17:13:12.174 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.759+0000 7f966e20d700 1 --2- 172.21.15.67:0/1441364265 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f965003e470 0x7f9650040930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:12.174 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.759+0000 7f966e20d700 1 --2- 172.21.15.67:0/1441364265 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f966810a650 0x7f966819cef0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:12.174 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.759+0000 7f966e20d700 1 -- 172.21.15.67:0/1441364265 >> 172.21.15.67:0/1441364265 conn(0x7f9668102b80 msgr2=0x7f96681035d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:12.174 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.759+0000 7f966e20d700 1 -- 172.21.15.67:0/1441364265 shutdown_connections 2024-05-03T17:13:12.174 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:10.759+0000 7f966e20d700 1 -- 172.21.15.67:0/1441364265 wait complete. 2024-05-03T17:13:12.174 INFO:teuthology.orchestra.run.smithi067.stdout:Fetching dashboard port number... 2024-05-03T17:13:13.186 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:12 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3228837322' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-03T17:13:13.469 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stdout 8443 2024-05-03T17:13:13.470 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.604+0000 7f98347e3700 1 Processor -- start 2024-05-03T17:13:13.470 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.605+0000 7f98347e3700 1 -- start start 2024-05-03T17:13:13.470 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.605+0000 7f98347e3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 0x7f982c10bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:13.470 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.605+0000 7f98347e3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f982c07d740 con 0x7f982c109990 2024-05-03T17:13:13.470 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.605+0000 7f983257f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 0x7f982c10bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:13.470 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.605+0000 7f983257f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 0x7f982c10bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42786/0 (socket says 172.21.15.67:42786) 2024-05-03T17:13:13.470 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.605+0000 7f983257f700 1 -- 172.21.15.67:0/3965262124 learned_addr learned my addr 172.21.15.67:0/3965262124 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:13.470 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.605+0000 7f983257f700 1 -- 172.21.15.67:0/3965262124 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982c10c2c0 con 0x7f982c109990 2024-05-03T17:13:13.470 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.605+0000 7f983257f700 1 --2- 172.21.15.67:0/3965262124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 0x7f982c10bd80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f981c0098a0 tx=0x7f981c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f09f89b935071 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:13.470 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.606+0000 7f983157d700 1 -- 172.21.15.67:0/3965262124 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f981c004160 con 0x7f982c109990 2024-05-03T17:13:13.470 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.606+0000 7f983157d700 1 -- 172.21.15.67:0/3965262124 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f981c004300 con 0x7f982c109990 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.606+0000 7f983157d700 1 -- 172.21.15.67:0/3965262124 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f981c011a20 con 0x7f982c109990 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.606+0000 7f98347e3700 1 -- 172.21.15.67:0/3965262124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 msgr2=0x7f982c10bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.606+0000 7f98347e3700 1 --2- 172.21.15.67:0/3965262124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 0x7f982c10bd80 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f981c0098a0 tx=0x7f981c009bb0 comp rx=0 tx=0).stop 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.606+0000 7f98347e3700 1 -- 172.21.15.67:0/3965262124 shutdown_connections 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.606+0000 7f98347e3700 1 --2- 172.21.15.67:0/3965262124 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 0x7f982c10bd80 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.606+0000 7f98347e3700 1 -- 172.21.15.67:0/3965262124 >> 172.21.15.67:0/3965262124 conn(0x7f982c1033e0 msgr2=0x7f982c105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.606+0000 7f98347e3700 1 -- 172.21.15.67:0/3965262124 shutdown_connections 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.606+0000 7f98347e3700 1 -- 172.21.15.67:0/3965262124 wait complete. 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.607+0000 7f98347e3700 1 Processor -- start 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.607+0000 7f98347e3700 1 -- start start 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.607+0000 7f98347e3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 0x7f982c19cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.607+0000 7f98347e3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f982c10d2c0 con 0x7f982c109990 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.607+0000 7f983257f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 0x7f982c19cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.607+0000 7f983257f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 0x7f982c19cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42790/0 (socket says 172.21.15.67:42790) 2024-05-03T17:13:13.471 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.607+0000 7f983257f700 1 -- 172.21.15.67:0/3228837322 learned_addr learned my addr 172.21.15.67:0/3228837322 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.608+0000 7f983257f700 1 -- 172.21.15.67:0/3228837322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f982c19d280 con 0x7f982c109990 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.608+0000 7f983257f700 1 --2- 172.21.15.67:0/3228837322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 0x7f982c19cd40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f981c004660 tx=0x7f981c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.608+0000 7f98237fe700 1 -- 172.21.15.67:0/3228837322 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f981c003aa0 con 0x7f982c109990 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.608+0000 7f98347e3700 1 -- 172.21.15.67:0/3228837322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f982c19d570 con 0x7f982c109990 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.608+0000 7f98237fe700 1 -- 172.21.15.67:0/3228837322 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f981c01b8f0 con 0x7f982c109990 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.608+0000 7f98237fe700 1 -- 172.21.15.67:0/3228837322 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f981c023900 con 0x7f982c109990 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.608+0000 7f98347e3700 1 -- 172.21.15.67:0/3228837322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f982c1a0200 con 0x7f982c109990 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.609+0000 7f98237fe700 1 -- 172.21.15.67:0/3228837322 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f981c023bc0 con 0x7f982c109990 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.609+0000 7f98347e3700 1 -- 172.21.15.67:0/3228837322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f982c064b00 con 0x7f982c109990 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.609+0000 7f98237fe700 1 --2- 172.21.15.67:0/3228837322 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f981803e4c0 0x7f9818040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.609+0000 7f98237fe700 1 -- 172.21.15.67:0/3228837322 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f981c054010 con 0x7f982c109990 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.609+0000 7f9831d7e700 1 --2- 172.21.15.67:0/3228837322 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f981803e4c0 0x7f9818040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.610+0000 7f9831d7e700 1 --2- 172.21.15.67:0/3228837322 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f981803e4c0 0x7f9818040980 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9828006fd0 tx=0x7f9828006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:13.472 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.614+0000 7f98237fe700 1 -- 172.21.15.67:0/3228837322 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f981c01f030 con 0x7f982c109990 2024-05-03T17:13:13.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.826+0000 7f98347e3700 1 -- 172.21.15.67:0/3228837322 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f982c1a03d0 con 0x7f982c109990 2024-05-03T17:13:13.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.826+0000 7f98237fe700 1 -- 172.21.15.67:0/3228837322 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f981c016260 con 0x7f982c109990 2024-05-03T17:13:13.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.828+0000 7f98347e3700 1 -- 172.21.15.67:0/3228837322 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f981803e4c0 msgr2=0x7f9818040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:13.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.828+0000 7f98347e3700 1 --2- 172.21.15.67:0/3228837322 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f981803e4c0 0x7f9818040980 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9828006fd0 tx=0x7f9828006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:13.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.828+0000 7f98347e3700 1 -- 172.21.15.67:0/3228837322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 msgr2=0x7f982c19cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:13.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.828+0000 7f98347e3700 1 --2- 172.21.15.67:0/3228837322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 0x7f982c19cd40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f981c004660 tx=0x7f981c011aa0 comp rx=0 tx=0).stop 2024-05-03T17:13:13.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.828+0000 7f98347e3700 1 -- 172.21.15.67:0/3228837322 shutdown_connections 2024-05-03T17:13:13.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.828+0000 7f98347e3700 1 --2- 172.21.15.67:0/3228837322 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f981803e4c0 0x7f9818040980 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:13.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.828+0000 7f98347e3700 1 --2- 172.21.15.67:0/3228837322 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f982c109990 0x7f982c19cd40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:13.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.828+0000 7f98347e3700 1 -- 172.21.15.67:0/3228837322 >> 172.21.15.67:0/3228837322 conn(0x7f982c1033e0 msgr2=0x7f982c103ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:13.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.828+0000 7f98347e3700 1 -- 172.21.15.67:0/3228837322 shutdown_connections 2024-05-03T17:13:13.473 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:12.828+0000 7f98347e3700 1 -- 172.21.15.67:0/3228837322 wait complete. 2024-05-03T17:13:13.477 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-03T17:13:13.477 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout disabled 2024-05-03T17:13:13.485 INFO:teuthology.orchestra.run.smithi067.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-03T17:13:13.485 INFO:teuthology.orchestra.run.smithi067.stdout:systemctl: stdout inactive 2024-05-03T17:13:13.485 INFO:teuthology.orchestra.run.smithi067.stdout:firewalld.service is not enabled 2024-05-03T17:13:13.485 INFO:teuthology.orchestra.run.smithi067.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-03T17:13:13.487 INFO:teuthology.orchestra.run.smithi067.stdout:Ceph Dashboard is now available at: 2024-05-03T17:13:13.487 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:13:13.487 INFO:teuthology.orchestra.run.smithi067.stdout: URL: https://smithi067:8443/ 2024-05-03T17:13:13.487 INFO:teuthology.orchestra.run.smithi067.stdout: User: admin 2024-05-03T17:13:13.487 INFO:teuthology.orchestra.run.smithi067.stdout: Password: 6uihlqaf3d 2024-05-03T17:13:13.487 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:13:13.487 INFO:teuthology.orchestra.run.smithi067.stdout:Saving cluster configuration to /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config directory 2024-05-03T17:13:13.488 INFO:teuthology.orchestra.run.smithi067.stdout:Enabling autotune for osd_memory_target 2024-05-03T17:13:14.132 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:13 smithi067 ceph-mon[29731]: mgrmap e13: x(active, since 6s) 2024-05-03T17:13:14.717 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.907+0000 7f1b43239700 1 Processor -- start 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.907+0000 7f1b43239700 1 -- start start 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.907+0000 7f1b43239700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 0x7f1b3c1081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.907+0000 7f1b43239700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3c1087b0 con 0x7f1b3c107dc0 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.908+0000 7f1b40fd5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 0x7f1b3c1081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.908+0000 7f1b40fd5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 0x7f1b3c1081e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42800/0 (socket says 172.21.15.67:42800) 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.908+0000 7f1b40fd5700 1 -- 172.21.15.67:0/4062153147 learned_addr learned my addr 172.21.15.67:0/4062153147 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.908+0000 7f1b40fd5700 1 -- 172.21.15.67:0/4062153147 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3c108fc0 con 0x7f1b3c107dc0 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.908+0000 7f1b40fd5700 1 --2- 172.21.15.67:0/4062153147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 0x7f1b3c1081e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00bb40 tx=0x7f1b2c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69acd1317b028173 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.908+0000 7f1b3b7fe700 1 -- 172.21.15.67:0/4062153147 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b2c00dbe0 con 0x7f1b3c107dc0 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.908+0000 7f1b3b7fe700 1 -- 172.21.15.67:0/4062153147 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b2c0044e0 con 0x7f1b3c107dc0 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.908+0000 7f1b3b7fe700 1 -- 172.21.15.67:0/4062153147 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b2c010470 con 0x7f1b3c107dc0 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.909+0000 7f1b43239700 1 -- 172.21.15.67:0/4062153147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 msgr2=0x7f1b3c1081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:14.718 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.909+0000 7f1b43239700 1 --2- 172.21.15.67:0/4062153147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 0x7f1b3c1081e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00bb40 tx=0x7f1b2c00be50 comp rx=0 tx=0).stop 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.909+0000 7f1b43239700 1 -- 172.21.15.67:0/4062153147 shutdown_connections 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.909+0000 7f1b43239700 1 --2- 172.21.15.67:0/4062153147 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 0x7f1b3c1081e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.909+0000 7f1b43239700 1 -- 172.21.15.67:0/4062153147 >> 172.21.15.67:0/4062153147 conn(0x7f1b3c103380 msgr2=0x7f1b3c1057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.909+0000 7f1b43239700 1 -- 172.21.15.67:0/4062153147 shutdown_connections 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.909+0000 7f1b43239700 1 -- 172.21.15.67:0/4062153147 wait complete. 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.910+0000 7f1b43239700 1 Processor -- start 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.910+0000 7f1b43239700 1 -- start start 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.910+0000 7f1b43239700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 0x7f1b3c19ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.910+0000 7f1b43239700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3c07bff0 con 0x7f1b3c107dc0 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.910+0000 7f1b40fd5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 0x7f1b3c19ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.910+0000 7f1b40fd5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 0x7f1b3c19ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42806/0 (socket says 172.21.15.67:42806) 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.910+0000 7f1b40fd5700 1 -- 172.21.15.67:0/858264596 learned_addr learned my addr 172.21.15.67:0/858264596 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.910+0000 7f1b40fd5700 1 -- 172.21.15.67:0/858264596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3c19f4b0 con 0x7f1b3c107dc0 2024-05-03T17:13:14.719 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.910+0000 7f1b40fd5700 1 --2- 172.21.15.67:0/858264596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 0x7f1b3c19ef70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00d650 tx=0x7f1b2c003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.911+0000 7f1b39ffb700 1 -- 172.21.15.67:0/858264596 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b2c010770 con 0x7f1b3c107dc0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.911+0000 7f1b43239700 1 -- 172.21.15.67:0/858264596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b3c19f740 con 0x7f1b3c107dc0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.911+0000 7f1b39ffb700 1 -- 172.21.15.67:0/858264596 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1b2c01d070 con 0x7f1b3c107dc0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.911+0000 7f1b39ffb700 1 -- 172.21.15.67:0/858264596 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b2c021bf0 con 0x7f1b3c107dc0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.911+0000 7f1b43239700 1 -- 172.21.15.67:0/858264596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b3c1a2430 con 0x7f1b3c107dc0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.912+0000 7f1b39ffb700 1 -- 172.21.15.67:0/858264596 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f1b2c01f030 con 0x7f1b3c107dc0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.912+0000 7f1b43239700 1 -- 172.21.15.67:0/858264596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b3c064b00 con 0x7f1b3c107dc0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.912+0000 7f1b39ffb700 1 --2- 172.21.15.67:0/858264596 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1b2403e510 0x7f1b240409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.912+0000 7f1b39ffb700 1 -- 172.21.15.67:0/858264596 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1b2c053b50 con 0x7f1b3c107dc0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.912+0000 7f1b3bfff700 1 --2- 172.21.15.67:0/858264596 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1b2403e510 0x7f1b240409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.913+0000 7f1b3bfff700 1 --2- 172.21.15.67:0/858264596 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1b2403e510 0x7f1b240409d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1b30006fd0 tx=0x7f1b30006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:13.917+0000 7f1b39ffb700 1 -- 172.21.15.67:0/858264596 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f1b2c015070 con 0x7f1b3c107dc0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:14.131+0000 7f1b43239700 1 -- 172.21.15.67:0/858264596 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f1b3c1940c0 con 0x7f1b3c107dc0 2024-05-03T17:13:14.720 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:14.131+0000 7f1b39ffb700 1 -- 172.21.15.67:0/858264596 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f1b3c1940c0 con 0x7f1b3c107dc0 2024-05-03T17:13:14.721 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:14.134+0000 7f1b43239700 1 -- 172.21.15.67:0/858264596 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1b2403e510 msgr2=0x7f1b240409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:14.721 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:14.134+0000 7f1b43239700 1 --2- 172.21.15.67:0/858264596 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1b2403e510 0x7f1b240409d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f1b30006fd0 tx=0x7f1b30006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:14.721 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:14.134+0000 7f1b43239700 1 -- 172.21.15.67:0/858264596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 msgr2=0x7f1b3c19ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:14.721 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:14.134+0000 7f1b43239700 1 --2- 172.21.15.67:0/858264596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 0x7f1b3c19ef70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c00d650 tx=0x7f1b2c003ff0 comp rx=0 tx=0).stop 2024-05-03T17:13:14.721 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:14.134+0000 7f1b43239700 1 -- 172.21.15.67:0/858264596 shutdown_connections 2024-05-03T17:13:14.721 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:14.134+0000 7f1b43239700 1 --2- 172.21.15.67:0/858264596 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1b2403e510 0x7f1b240409d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:14.721 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:14.134+0000 7f1b43239700 1 --2- 172.21.15.67:0/858264596 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b3c107dc0 0x7f1b3c19ef70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:14.721 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:14.134+0000 7f1b43239700 1 -- 172.21.15.67:0/858264596 >> 172.21.15.67:0/858264596 conn(0x7f1b3c103380 msgr2=0x7f1b3c191180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:14.721 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:14.134+0000 7f1b43239700 1 -- 172.21.15.67:0/858264596 shutdown_connections 2024-05-03T17:13:14.721 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:14.134+0000 7f1b43239700 1 -- 172.21.15.67:0/858264596 wait complete. 2024-05-03T17:13:16.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.160+0000 7f531749a700 1 Processor -- start 2024-05-03T17:13:16.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.160+0000 7f531749a700 1 -- start start 2024-05-03T17:13:16.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.160+0000 7f531749a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 0x7f5310108230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:16.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.160+0000 7f531749a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5310108800 con 0x7f5310107e10 2024-05-03T17:13:16.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.160+0000 7f5315236700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 0x7f5310108230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:16.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.161+0000 7f5315236700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 0x7f5310108230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42812/0 (socket says 172.21.15.67:42812) 2024-05-03T17:13:16.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.161+0000 7f5315236700 1 -- 172.21.15.67:0/1906184558 learned_addr learned my addr 172.21.15.67:0/1906184558 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:16.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.161+0000 7f5315236700 1 -- 172.21.15.67:0/1906184558 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5310108fc0 con 0x7f5310107e10 2024-05-03T17:13:16.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.161+0000 7f5315236700 1 --2- 172.21.15.67:0/1906184558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 0x7f5310108230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f530c0098a0 tx=0x7f530c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e811f55446e133 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:16.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.161+0000 7f5307fff700 1 -- 172.21.15.67:0/1906184558 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f530c004160 con 0x7f5310107e10 2024-05-03T17:13:16.108 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.161+0000 7f5307fff700 1 -- 172.21.15.67:0/1906184558 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f530c004300 con 0x7f5310107e10 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.161+0000 7f5307fff700 1 -- 172.21.15.67:0/1906184558 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f530c011a20 con 0x7f5310107e10 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.162+0000 7f531749a700 1 -- 172.21.15.67:0/1906184558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 msgr2=0x7f5310108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.162+0000 7f531749a700 1 --2- 172.21.15.67:0/1906184558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 0x7f5310108230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f530c0098a0 tx=0x7f530c009bb0 comp rx=0 tx=0).stop 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.162+0000 7f531749a700 1 -- 172.21.15.67:0/1906184558 shutdown_connections 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.162+0000 7f531749a700 1 --2- 172.21.15.67:0/1906184558 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 0x7f5310108230 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.162+0000 7f531749a700 1 -- 172.21.15.67:0/1906184558 >> 172.21.15.67:0/1906184558 conn(0x7f5310103390 msgr2=0x7f53101057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.162+0000 7f531749a700 1 -- 172.21.15.67:0/1906184558 shutdown_connections 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.162+0000 7f531749a700 1 -- 172.21.15.67:0/1906184558 wait complete. 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.162+0000 7f531749a700 1 Processor -- start 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.163+0000 7f531749a700 1 -- start start 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.163+0000 7f531749a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 0x7f531007d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.163+0000 7f531749a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f531010c370 con 0x7f5310107e10 2024-05-03T17:13:16.109 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.163+0000 7f5315236700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 0x7f531007d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.163+0000 7f5315236700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 0x7f531007d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42814/0 (socket says 172.21.15.67:42814) 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.163+0000 7f5315236700 1 -- 172.21.15.67:0/1191281728 learned_addr learned my addr 172.21.15.67:0/1191281728 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.163+0000 7f5315236700 1 -- 172.21.15.67:0/1191281728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f531019fb00 con 0x7f5310107e10 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.163+0000 7f5315236700 1 --2- 172.21.15.67:0/1191281728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 0x7f531007d300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f530c000c00 tx=0x7f530c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.163+0000 7f53067fc700 1 -- 172.21.15.67:0/1191281728 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f530c003aa0 con 0x7f5310107e10 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.163+0000 7f53067fc700 1 -- 172.21.15.67:0/1191281728 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f530c01b8f0 con 0x7f5310107e10 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.163+0000 7f531749a700 1 -- 172.21.15.67:0/1191281728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f531019fcd0 con 0x7f5310107e10 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.163+0000 7f53067fc700 1 -- 172.21.15.67:0/1191281728 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f530c023890 con 0x7f5310107e10 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.164+0000 7f531749a700 1 -- 172.21.15.67:0/1191281728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53101a0070 con 0x7f5310107e10 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.164+0000 7f531749a700 1 -- 172.21.15.67:0/1191281728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5310064b00 con 0x7f5310107e10 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.165+0000 7f53067fc700 1 -- 172.21.15.67:0/1191281728 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f530c01b440 con 0x7f5310107e10 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.165+0000 7f53067fc700 1 --2- 172.21.15.67:0/1191281728 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f52fc03e560 0x7f52fc040a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:16.110 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.165+0000 7f53067fc700 1 -- 172.21.15.67:0/1191281728 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f530c054020 con 0x7f5310107e10 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.165+0000 7f5314a35700 1 --2- 172.21.15.67:0/1191281728 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f52fc03e560 0x7f52fc040a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.165+0000 7f5314a35700 1 --2- 172.21.15.67:0/1191281728 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f52fc03e560 0x7f52fc040a20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5300006fd0 tx=0x7f5300006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.170+0000 7f53067fc700 1 -- 172.21.15.67:0/1191281728 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f530c01f030 con 0x7f5310107e10 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.500+0000 7f531749a700 1 -- 172.21.15.67:0/1191281728 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f53101a0390 con 0x7f5310107e10 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.503+0000 7f53067fc700 1 -- 172.21.15.67:0/1191281728 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f530c017e90 con 0x7f5310107e10 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.505+0000 7f531749a700 1 -- 172.21.15.67:0/1191281728 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f52fc03e560 msgr2=0x7f52fc040a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.505+0000 7f531749a700 1 --2- 172.21.15.67:0/1191281728 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f52fc03e560 0x7f52fc040a20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5300006fd0 tx=0x7f5300006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.505+0000 7f531749a700 1 -- 172.21.15.67:0/1191281728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 msgr2=0x7f531007d300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.505+0000 7f531749a700 1 --2- 172.21.15.67:0/1191281728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 0x7f531007d300 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f530c000c00 tx=0x7f530c011aa0 comp rx=0 tx=0).stop 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.505+0000 7f531749a700 1 -- 172.21.15.67:0/1191281728 shutdown_connections 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.505+0000 7f531749a700 1 --2- 172.21.15.67:0/1191281728 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f52fc03e560 0x7f52fc040a20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.505+0000 7f531749a700 1 --2- 172.21.15.67:0/1191281728 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5310107e10 0x7f531007d300 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.505+0000 7f531749a700 1 -- 172.21.15.67:0/1191281728 >> 172.21.15.67:0/1191281728 conn(0x7f5310103390 msgr2=0x7f5310103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.505+0000 7f531749a700 1 -- 172.21.15.67:0/1191281728 shutdown_connections 2024-05-03T17:13:16.111 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr 2024-05-03T17:13:15.505+0000 7f531749a700 1 -- 172.21.15.67:0/1191281728 wait complete. 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout:Or, if you are only running a single cluster on this host: 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: ceph telemetry on 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout:For more information see: 2024-05-03T17:13:16.112 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:13:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-03T17:13:16.113 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:13:16.113 INFO:teuthology.orchestra.run.smithi067.stdout:Bootstrap complete. 2024-05-03T17:13:16.150 INFO:tasks.cephadm:Fetching config... 2024-05-03T17:13:16.151 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:13:16.151 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-03T17:13:16.167 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-03T17:13:16.167 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:13:16.167 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-03T17:13:16.223 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-03T17:13:16.223 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:13:16.223 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-03T17:13:16.289 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-03T17:13:16.289 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:13:16.289 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-03T17:13:16.345 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-03T17:13:16.345 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmOjo8dWltEJGOxPvkwh30otUWA1zjSWL+5hoobQWmhu5Z8xaYm8AtC9gtK/xBWW2qTOJU9BrZFPiTcqEAtUakMHGxX2+pvmQhZSwpo8ATkZFgn9RGEDeySGbIjpf/fr38vthQtE3m8XMy2j1oPcRwBb+Mu5/E7knhIiZ7eezhcsqF2TKxIZ/Oke6KYb9FsmmKvQIWXj/QStz4/SXrvk6HNIKH+MSd6vaMY3XERNi4cUvv6xpVeJyLjTbKodX0y1ERrLUxkwBX8E8k04WvLc357Wu8ENekAXyZZMf5nIyN2CJ+BLEv+8vpFSfAv+KNrizjfNx/vj+ffQ8dJeppZk3+3cGoNpmihivl5SE1/7nK8F/ClXnIMSEJ5QlsWw6txNbMXZyt5Kw3Rs3HRvYVpQs1b9fr/51ogeT7hpFZrRWptNyHcfESWGtrw7tMbIEzeZC5eDkjyKQB+jGezSO/hz1LkRs7uyibJfivDq18XWVAjw7gHCtth5+E4k7z0Di+mE= ceph-20c8e22e-0970-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-03T17:13:16.418 INFO:teuthology.orchestra.run.smithi067.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmOjo8dWltEJGOxPvkwh30otUWA1zjSWL+5hoobQWmhu5Z8xaYm8AtC9gtK/xBWW2qTOJU9BrZFPiTcqEAtUakMHGxX2+pvmQhZSwpo8ATkZFgn9RGEDeySGbIjpf/fr38vthQtE3m8XMy2j1oPcRwBb+Mu5/E7knhIiZ7eezhcsqF2TKxIZ/Oke6KYb9FsmmKvQIWXj/QStz4/SXrvk6HNIKH+MSd6vaMY3XERNi4cUvv6xpVeJyLjTbKodX0y1ERrLUxkwBX8E8k04WvLc357Wu8ENekAXyZZMf5nIyN2CJ+BLEv+8vpFSfAv+KNrizjfNx/vj+ffQ8dJeppZk3+3cGoNpmihivl5SE1/7nK8F/ClXnIMSEJ5QlsWw6txNbMXZyt5Kw3Rs3HRvYVpQs1b9fr/51ogeT7hpFZrRWptNyHcfESWGtrw7tMbIEzeZC5eDkjyKQB+jGezSO/hz1LkRs7uyibJfivDq18XWVAjw7gHCtth5+E4k7z0Di+mE= ceph-20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:13:16.429 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmOjo8dWltEJGOxPvkwh30otUWA1zjSWL+5hoobQWmhu5Z8xaYm8AtC9gtK/xBWW2qTOJU9BrZFPiTcqEAtUakMHGxX2+pvmQhZSwpo8ATkZFgn9RGEDeySGbIjpf/fr38vthQtE3m8XMy2j1oPcRwBb+Mu5/E7knhIiZ7eezhcsqF2TKxIZ/Oke6KYb9FsmmKvQIWXj/QStz4/SXrvk6HNIKH+MSd6vaMY3XERNi4cUvv6xpVeJyLjTbKodX0y1ERrLUxkwBX8E8k04WvLc357Wu8ENekAXyZZMf5nIyN2CJ+BLEv+8vpFSfAv+KNrizjfNx/vj+ffQ8dJeppZk3+3cGoNpmihivl5SE1/7nK8F/ClXnIMSEJ5QlsWw6txNbMXZyt5Kw3Rs3HRvYVpQs1b9fr/51ogeT7hpFZrRWptNyHcfESWGtrw7tMbIEzeZC5eDkjyKQB+jGezSO/hz1LkRs7uyibJfivDq18XWVAjw7gHCtth5+E4k7z0Di+mE= ceph-20c8e22e-0970-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-03T17:13:16.469 INFO:teuthology.orchestra.run.smithi070.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmOjo8dWltEJGOxPvkwh30otUWA1zjSWL+5hoobQWmhu5Z8xaYm8AtC9gtK/xBWW2qTOJU9BrZFPiTcqEAtUakMHGxX2+pvmQhZSwpo8ATkZFgn9RGEDeySGbIjpf/fr38vthQtE3m8XMy2j1oPcRwBb+Mu5/E7knhIiZ7eezhcsqF2TKxIZ/Oke6KYb9FsmmKvQIWXj/QStz4/SXrvk6HNIKH+MSd6vaMY3XERNi4cUvv6xpVeJyLjTbKodX0y1ERrLUxkwBX8E8k04WvLc357Wu8ENekAXyZZMf5nIyN2CJ+BLEv+8vpFSfAv+KNrizjfNx/vj+ffQ8dJeppZk3+3cGoNpmihivl5SE1/7nK8F/ClXnIMSEJ5QlsWw6txNbMXZyt5Kw3Rs3HRvYVpQs1b9fr/51ogeT7hpFZrRWptNyHcfESWGtrw7tMbIEzeZC5eDkjyKQB+jGezSO/hz1LkRs7uyibJfivDq18XWVAjw7gHCtth5+E4k7z0Di+mE= ceph-20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:13:16.480 DEBUG:teuthology.orchestra.run.smithi082:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmOjo8dWltEJGOxPvkwh30otUWA1zjSWL+5hoobQWmhu5Z8xaYm8AtC9gtK/xBWW2qTOJU9BrZFPiTcqEAtUakMHGxX2+pvmQhZSwpo8ATkZFgn9RGEDeySGbIjpf/fr38vthQtE3m8XMy2j1oPcRwBb+Mu5/E7knhIiZ7eezhcsqF2TKxIZ/Oke6KYb9FsmmKvQIWXj/QStz4/SXrvk6HNIKH+MSd6vaMY3XERNi4cUvv6xpVeJyLjTbKodX0y1ERrLUxkwBX8E8k04WvLc357Wu8ENekAXyZZMf5nIyN2CJ+BLEv+8vpFSfAv+KNrizjfNx/vj+ffQ8dJeppZk3+3cGoNpmihivl5SE1/7nK8F/ClXnIMSEJ5QlsWw6txNbMXZyt5Kw3Rs3HRvYVpQs1b9fr/51ogeT7hpFZrRWptNyHcfESWGtrw7tMbIEzeZC5eDkjyKQB+jGezSO/hz1LkRs7uyibJfivDq18XWVAjw7gHCtth5+E4k7z0Di+mE= ceph-20c8e22e-0970-11ef-bc97-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-03T17:13:16.519 INFO:teuthology.orchestra.run.smithi082.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSmOjo8dWltEJGOxPvkwh30otUWA1zjSWL+5hoobQWmhu5Z8xaYm8AtC9gtK/xBWW2qTOJU9BrZFPiTcqEAtUakMHGxX2+pvmQhZSwpo8ATkZFgn9RGEDeySGbIjpf/fr38vthQtE3m8XMy2j1oPcRwBb+Mu5/E7knhIiZ7eezhcsqF2TKxIZ/Oke6KYb9FsmmKvQIWXj/QStz4/SXrvk6HNIKH+MSd6vaMY3XERNi4cUvv6xpVeJyLjTbKodX0y1ERrLUxkwBX8E8k04WvLc357Wu8ENekAXyZZMf5nIyN2CJ+BLEv+8vpFSfAv+KNrizjfNx/vj+ffQ8dJeppZk3+3cGoNpmihivl5SE1/7nK8F/ClXnIMSEJ5QlsWw6txNbMXZyt5Kw3Rs3HRvYVpQs1b9fr/51ogeT7hpFZrRWptNyHcfESWGtrw7tMbIEzeZC5eDkjyKQB+jGezSO/hz1LkRs7uyibJfivDq18XWVAjw7gHCtth5+E4k7z0Di+mE= ceph-20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:13:16.530 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-03T17:13:16.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:16 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1191281728' entity='client.admin' 2024-05-03T17:13:16.774 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:13:19.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.467+0000 7f48021b7700 1 -- 172.21.15.67:0/2269605380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc1039f0 msgr2=0x7f47fc105de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:19.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.467+0000 7f48021b7700 1 --2- 172.21.15.67:0/2269605380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc1039f0 0x7f47fc105de0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f47ec009910 tx=0x7f47ec009c20 comp rx=0 tx=0).stop 2024-05-03T17:13:19.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.467+0000 7f48021b7700 1 -- 172.21.15.67:0/2269605380 shutdown_connections 2024-05-03T17:13:19.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.467+0000 7f48021b7700 1 --2- 172.21.15.67:0/2269605380 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc1039f0 0x7f47fc105de0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:19.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.467+0000 7f48021b7700 1 -- 172.21.15.67:0/2269605380 >> 172.21.15.67:0/2269605380 conn(0x7f47fc0fd3a0 msgr2=0x7f47fc0ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:19.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.467+0000 7f48021b7700 1 -- 172.21.15.67:0/2269605380 shutdown_connections 2024-05-03T17:13:19.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.467+0000 7f48021b7700 1 -- 172.21.15.67:0/2269605380 wait complete. 2024-05-03T17:13:19.471 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.468+0000 7f48021b7700 1 Processor -- start 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.468+0000 7f48021b7700 1 -- start start 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.468+0000 7f48021b7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc1039f0 0x7f47fc103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.468+0000 7f48021b7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc108e50 con 0x7f47fc1039f0 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.468+0000 7f47fb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc1039f0 0x7f47fc103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.468+0000 7f47fb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc1039f0 0x7f47fc103270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57138/0 (socket says 172.21.15.67:57138) 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.468+0000 7f47fb7fe700 1 -- 172.21.15.67:0/4030502299 learned_addr learned my addr 172.21.15.67:0/4030502299 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.469+0000 7f47fb7fe700 1 -- 172.21.15.67:0/4030502299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc1037b0 con 0x7f47fc1039f0 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.469+0000 7f47fb7fe700 1 --2- 172.21.15.67:0/4030502299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc1039f0 0x7f47fc103270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f47ec005190 tx=0x7f47ec004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.469+0000 7f47f8ff9700 1 -- 172.21.15.67:0/4030502299 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47ec027070 con 0x7f47fc1039f0 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.469+0000 7f47f8ff9700 1 -- 172.21.15.67:0/4030502299 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f47ec022470 con 0x7f47fc1039f0 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.469+0000 7f48021b7700 1 -- 172.21.15.67:0/4030502299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47fc1019a0 con 0x7f47fc1039f0 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.469+0000 7f47f8ff9700 1 -- 172.21.15.67:0/4030502299 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f47ec00f5f0 con 0x7f47fc1039f0 2024-05-03T17:13:19.472 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.469+0000 7f48021b7700 1 -- 172.21.15.67:0/4030502299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47fc101ee0 con 0x7f47fc1039f0 2024-05-03T17:13:19.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.470+0000 7f47f8ff9700 1 -- 172.21.15.67:0/4030502299 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f47ec02a070 con 0x7f47fc1039f0 2024-05-03T17:13:19.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.470+0000 7f48021b7700 1 -- 172.21.15.67:0/4030502299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47fc1095d0 con 0x7f47fc1039f0 2024-05-03T17:13:19.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.470+0000 7f47f8ff9700 1 --2- 172.21.15.67:0/4030502299 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f47e403e510 0x7f47e40409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:19.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.470+0000 7f47f8ff9700 1 -- 172.21.15.67:0/4030502299 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f47ec01e030 con 0x7f47fc1039f0 2024-05-03T17:13:19.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.470+0000 7f47faffd700 1 --2- 172.21.15.67:0/4030502299 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f47e403e510 0x7f47e40409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:19.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.471+0000 7f47faffd700 1 --2- 172.21.15.67:0/4030502299 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f47e403e510 0x7f47e40409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f47f0006fd0 tx=0x7f47f0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:19.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.476+0000 7f47f8ff9700 1 -- 172.21.15.67:0/4030502299 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f47ec013070 con 0x7f47fc1039f0 2024-05-03T17:13:19.694 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.692+0000 7f48021b7700 1 -- 172.21.15.67:0/4030502299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f47fc1918d0 con 0x7f47fc1039f0 2024-05-03T17:13:19.696 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.695+0000 7f47f8ff9700 1 -- 172.21.15.67:0/4030502299 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f47ec0310c0 con 0x7f47fc1039f0 2024-05-03T17:13:19.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.697+0000 7f48021b7700 1 -- 172.21.15.67:0/4030502299 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f47e403e510 msgr2=0x7f47e40409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:19.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.697+0000 7f48021b7700 1 --2- 172.21.15.67:0/4030502299 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f47e403e510 0x7f47e40409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f47f0006fd0 tx=0x7f47f0006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:19.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.697+0000 7f48021b7700 1 -- 172.21.15.67:0/4030502299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc1039f0 msgr2=0x7f47fc103270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:19.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.697+0000 7f48021b7700 1 --2- 172.21.15.67:0/4030502299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc1039f0 0x7f47fc103270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f47ec005190 tx=0x7f47ec004f80 comp rx=0 tx=0).stop 2024-05-03T17:13:19.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.697+0000 7f48021b7700 1 -- 172.21.15.67:0/4030502299 shutdown_connections 2024-05-03T17:13:19.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.697+0000 7f48021b7700 1 --2- 172.21.15.67:0/4030502299 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f47e403e510 0x7f47e40409d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:19.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.697+0000 7f48021b7700 1 --2- 172.21.15.67:0/4030502299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f47fc1039f0 0x7f47fc103270 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:19.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.697+0000 7f48021b7700 1 -- 172.21.15.67:0/4030502299 >> 172.21.15.67:0/4030502299 conn(0x7f47fc0fd3a0 msgr2=0x7f47fc0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:19.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.697+0000 7f48021b7700 1 -- 172.21.15.67:0/4030502299 shutdown_connections 2024-05-03T17:13:19.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:19.698+0000 7f48021b7700 1 -- 172.21.15.67:0/4030502299 wait complete. 2024-05-03T17:13:20.295 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-03T17:13:20.295 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-03T17:13:20.772 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:13:20.928 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:20 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/4030502299' entity='client.admin' 2024-05-03T17:13:21.994 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:21.994 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:21.994 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi067", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:13:21.994 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:21.994 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:13:21.994 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.354+0000 7f6b86c8d700 1 -- 172.21.15.67:0/198633850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b80104570 msgr2=0x7f6b80104990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.354+0000 7f6b86c8d700 1 --2- 172.21.15.67:0/198633850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b80104570 0x7f6b80104990 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6b74009910 tx=0x7f6b74009c20 comp rx=0 tx=0).stop 2024-05-03T17:13:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.354+0000 7f6b86c8d700 1 -- 172.21.15.67:0/198633850 shutdown_connections 2024-05-03T17:13:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.354+0000 7f6b86c8d700 1 --2- 172.21.15.67:0/198633850 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b80104570 0x7f6b80104990 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.354+0000 7f6b86c8d700 1 -- 172.21.15.67:0/198633850 >> 172.21.15.67:0/198633850 conn(0x7f6b800fe4f0 msgr2=0x7f6b80100910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:22.357 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.354+0000 7f6b86c8d700 1 -- 172.21.15.67:0/198633850 shutdown_connections 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.354+0000 7f6b86c8d700 1 -- 172.21.15.67:0/198633850 wait complete. 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.354+0000 7f6b86c8d700 1 Processor -- start 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.354+0000 7f6b86c8d700 1 -- start start 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.354+0000 7f6b86c8d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b80104570 0x7f6b8019a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.354+0000 7f6b86c8d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6b80108ef0 con 0x7f6b80104570 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.355+0000 7f6b84a29700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b80104570 0x7f6b8019a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.355+0000 7f6b84a29700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b80104570 0x7f6b8019a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57152/0 (socket says 172.21.15.67:57152) 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.355+0000 7f6b84a29700 1 -- 172.21.15.67:0/1395038437 learned_addr learned my addr 172.21.15.67:0/1395038437 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.355+0000 7f6b84a29700 1 -- 172.21.15.67:0/1395038437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b8019adf0 con 0x7f6b80104570 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.355+0000 7f6b84a29700 1 --2- 172.21.15.67:0/1395038437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b80104570 0x7f6b8019a8b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6b74005b40 tx=0x7f6b74004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.355+0000 7f6b7dffb700 1 -- 172.21.15.67:0/1395038437 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b74027070 con 0x7f6b80104570 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.355+0000 7f6b7dffb700 1 -- 172.21.15.67:0/1395038437 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b7400be10 con 0x7f6b80104570 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.355+0000 7f6b7dffb700 1 -- 172.21.15.67:0/1395038437 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6b74022de0 con 0x7f6b80104570 2024-05-03T17:13:22.358 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.355+0000 7f6b86c8d700 1 -- 172.21.15.67:0/1395038437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b8019b080 con 0x7f6b80104570 2024-05-03T17:13:22.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.355+0000 7f6b86c8d700 1 -- 172.21.15.67:0/1395038437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b8019dd70 con 0x7f6b80104570 2024-05-03T17:13:22.359 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.356+0000 7f6b7dffb700 1 -- 172.21.15.67:0/1395038437 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f6b7402a070 con 0x7f6b80104570 2024-05-03T17:13:22.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.356+0000 7f6b86c8d700 1 -- 172.21.15.67:0/1395038437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b80109670 con 0x7f6b80104570 2024-05-03T17:13:22.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.356+0000 7f6b7dffb700 1 --2- 172.21.15.67:0/1395038437 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f6b7003e510 0x7f6b700409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:22.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.357+0000 7f6b7dffb700 1 -- 172.21.15.67:0/1395038437 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6b7401e030 con 0x7f6b80104570 2024-05-03T17:13:22.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.357+0000 7f6b7ffff700 1 --2- 172.21.15.67:0/1395038437 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f6b7003e510 0x7f6b700409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:22.360 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.357+0000 7f6b7ffff700 1 --2- 172.21.15.67:0/1395038437 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f6b7003e510 0x7f6b700409d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c006fd0 tx=0x7f6b6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:22.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.362+0000 7f6b7dffb700 1 -- 172.21.15.67:0/1395038437 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6b74013070 con 0x7f6b80104570 2024-05-03T17:13:22.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.577+0000 7f6b86c8d700 1 -- 172.21.15.67:0/1395038437 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f6b800638e0 con 0x7f6b7003e510 2024-05-03T17:13:22.582 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.581+0000 7f6b7dffb700 1 -- 172.21.15.67:0/1395038437 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6b800638e0 con 0x7f6b7003e510 2024-05-03T17:13:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.583+0000 7f6b86c8d700 1 -- 172.21.15.67:0/1395038437 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f6b7003e510 msgr2=0x7f6b700409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:22.584 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.583+0000 7f6b86c8d700 1 --2- 172.21.15.67:0/1395038437 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f6b7003e510 0x7f6b700409d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6b6c006fd0 tx=0x7f6b6c006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.583+0000 7f6b86c8d700 1 -- 172.21.15.67:0/1395038437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b80104570 msgr2=0x7f6b8019a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.583+0000 7f6b86c8d700 1 --2- 172.21.15.67:0/1395038437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b80104570 0x7f6b8019a8b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6b74005b40 tx=0x7f6b74004f80 comp rx=0 tx=0).stop 2024-05-03T17:13:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.583+0000 7f6b86c8d700 1 -- 172.21.15.67:0/1395038437 shutdown_connections 2024-05-03T17:13:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.583+0000 7f6b86c8d700 1 --2- 172.21.15.67:0/1395038437 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f6b7003e510 0x7f6b700409d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.583+0000 7f6b86c8d700 1 --2- 172.21.15.67:0/1395038437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6b80104570 0x7f6b8019a8b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.583+0000 7f6b86c8d700 1 -- 172.21.15.67:0/1395038437 >> 172.21.15.67:0/1395038437 conn(0x7f6b800fe4f0 msgr2=0x7f6b800ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.583+0000 7f6b86c8d700 1 -- 172.21.15.67:0/1395038437 shutdown_connections 2024-05-03T17:13:22.585 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:22.583+0000 7f6b86c8d700 1 -- 172.21.15.67:0/1395038437 wait complete. 2024-05-03T17:13:23.357 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi070 2024-05-03T17:13:23.357 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-05-03T17:13:23.357 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/etc/ceph/ceph.conf 2024-05-03T17:13:23.376 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-05-03T17:13:23.376 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T17:13:23.432 INFO:tasks.cephadm:Adding host smithi070 to orchestrator... 2024-05-03T17:13:23.432 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch host add smithi070 2024-05-03T17:13:23.679 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:13:23.919 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:23 smithi067 ceph-mon[29731]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:13:23.919 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:23.919 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:13:23.919 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:13:23.919 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:13:23.919 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:23 smithi067 ceph-mon[29731]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-03T17:13:23.919 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:23 smithi067 ceph-mon[29731]: Updating smithi067:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:13:23.919 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:23.919 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:23.919 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:24 smithi067 ceph-mon[29731]: Updating smithi067:/etc/ceph/ceph.client.admin.keyring 2024-05-03T17:13:24.725 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:24 smithi067 ceph-mon[29731]: Updating smithi067:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-03T17:13:25.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.193+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/2834956470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78102900 msgr2=0x7f8e78102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:25.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.193+0000 7f8e7fe3f700 1 --2- 172.21.15.67:0/2834956470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78102900 0x7f8e78102ce0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8e6800b0a0 tx=0x7f8e6800b3b0 comp rx=0 tx=0).stop 2024-05-03T17:13:25.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.193+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/2834956470 shutdown_connections 2024-05-03T17:13:25.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.193+0000 7f8e7fe3f700 1 --2- 172.21.15.67:0/2834956470 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78102900 0x7f8e78102ce0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:25.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.193+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/2834956470 >> 172.21.15.67:0/2834956470 conn(0x7f8e78079330 msgr2=0x7f8e78079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:25.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.193+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/2834956470 shutdown_connections 2024-05-03T17:13:25.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.193+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/2834956470 wait complete. 2024-05-03T17:13:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.194+0000 7f8e7fe3f700 1 Processor -- start 2024-05-03T17:13:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.194+0000 7f8e7fe3f700 1 -- start start 2024-05-03T17:13:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.194+0000 7f8e7fe3f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78102900 0x7f8e7819d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.194+0000 7f8e7fe3f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e7810c310 con 0x7f8e78102900 2024-05-03T17:13:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.194+0000 7f8e7dbdb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78102900 0x7f8e7819d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.194+0000 7f8e7dbdb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78102900 0x7f8e7819d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57170/0 (socket says 172.21.15.67:57170) 2024-05-03T17:13:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.194+0000 7f8e7dbdb700 1 -- 172.21.15.67:0/740386949 learned_addr learned my addr 172.21.15.67:0/740386949 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.195+0000 7f8e7dbdb700 1 -- 172.21.15.67:0/740386949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7819d980 con 0x7f8e78102900 2024-05-03T17:13:25.197 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.195+0000 7f8e7dbdb700 1 --2- 172.21.15.67:0/740386949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78102900 0x7f8e7819d440 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8e68006130 tx=0x7f8e680097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.195+0000 7f8e76ffd700 1 -- 172.21.15.67:0/740386949 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e68021020 con 0x7f8e78102900 2024-05-03T17:13:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.195+0000 7f8e76ffd700 1 -- 172.21.15.67:0/740386949 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e680079b0 con 0x7f8e78102900 2024-05-03T17:13:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.195+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/740386949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e7819dc10 con 0x7f8e78102900 2024-05-03T17:13:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.195+0000 7f8e76ffd700 1 -- 172.21.15.67:0/740386949 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8e68017040 con 0x7f8e78102900 2024-05-03T17:13:25.198 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.195+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/740386949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e78197600 con 0x7f8e78102900 2024-05-03T17:13:25.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.196+0000 7f8e76ffd700 1 -- 172.21.15.67:0/740386949 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f8e68024020 con 0x7f8e78102900 2024-05-03T17:13:25.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.196+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/740386949 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e781036b0 con 0x7f8e78102900 2024-05-03T17:13:25.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.196+0000 7f8e76ffd700 1 --2- 172.21.15.67:0/740386949 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8e6003e510 0x7f8e600409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:25.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.196+0000 7f8e76ffd700 1 -- 172.21.15.67:0/740386949 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f8e6800f050 con 0x7f8e78102900 2024-05-03T17:13:25.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.196+0000 7f8e7d3da700 1 --2- 172.21.15.67:0/740386949 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8e6003e510 0x7f8e600409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:25.199 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.197+0000 7f8e7d3da700 1 --2- 172.21.15.67:0/740386949 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8e6003e510 0x7f8e600409d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c006fd0 tx=0x7f8e6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:25.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.201+0000 7f8e76ffd700 1 -- 172.21.15.67:0/740386949 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f8e6802b020 con 0x7f8e78102900 2024-05-03T17:13:25.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:25.419+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/740386949 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi070", "target": ["mon-mgr", ""]}) -- 0x7f8e78064b00 con 0x7f8e6003e510 2024-05-03T17:13:26.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:26 smithi067 ceph-mon[29731]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi070", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:13:27.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:27 smithi067 ceph-mon[29731]: Deploying cephadm binary to smithi070 2024-05-03T17:13:27.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:27 smithi067 ceph-mon[29731]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:29.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:29.684+0000 7f8e76ffd700 1 -- 172.21.15.67:0/740386949 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f8e78064b00 con 0x7f8e6003e510 2024-05-03T17:13:29.686 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:29 smithi067 ceph-mon[29731]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:29.687 INFO:teuthology.orchestra.run.smithi067.stdout:Added host 'smithi070' with addr '172.21.15.70' 2024-05-03T17:13:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:29.687+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/740386949 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8e6003e510 msgr2=0x7f8e600409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:29.687+0000 7f8e7fe3f700 1 --2- 172.21.15.67:0/740386949 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8e6003e510 0x7f8e600409d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c006fd0 tx=0x7f8e6c006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:29.687+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/740386949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78102900 msgr2=0x7f8e7819d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:29.687+0000 7f8e7fe3f700 1 --2- 172.21.15.67:0/740386949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78102900 0x7f8e7819d440 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8e68006130 tx=0x7f8e680097d0 comp rx=0 tx=0).stop 2024-05-03T17:13:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:29.687+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/740386949 shutdown_connections 2024-05-03T17:13:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:29.687+0000 7f8e7fe3f700 1 --2- 172.21.15.67:0/740386949 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8e6003e510 0x7f8e600409d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:29.687+0000 7f8e7fe3f700 1 --2- 172.21.15.67:0/740386949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e78102900 0x7f8e7819d440 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:29.687+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/740386949 >> 172.21.15.67:0/740386949 conn(0x7f8e78079330 msgr2=0x7f8e78103a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:29.687+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/740386949 shutdown_connections 2024-05-03T17:13:29.689 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:29.687+0000 7f8e7fe3f700 1 -- 172.21.15.67:0/740386949 wait complete. 2024-05-03T17:13:30.317 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch host ls --format=json 2024-05-03T17:13:30.567 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:13:30.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:30 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:30.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:30 smithi067 ceph-mon[29731]: Added host smithi070 2024-05-03T17:13:30.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:30 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:13:30.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:30 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:31.925 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:31 smithi067 ceph-mon[29731]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:31.925 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.161+0000 7fa0ba80d700 1 -- 172.21.15.67:0/1076748536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b41040b0 msgr2=0x7fa0b4104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.161+0000 7fa0ba80d700 1 --2- 172.21.15.67:0/1076748536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b41040b0 0x7fa0b4104490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4009910 tx=0x7fa0a4009c20 comp rx=0 tx=0).stop 2024-05-03T17:13:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.161+0000 7fa0ba80d700 1 -- 172.21.15.67:0/1076748536 shutdown_connections 2024-05-03T17:13:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.161+0000 7fa0ba80d700 1 --2- 172.21.15.67:0/1076748536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b41040b0 0x7fa0b4104490 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.161+0000 7fa0ba80d700 1 -- 172.21.15.67:0/1076748536 >> 172.21.15.67:0/1076748536 conn(0x7fa0b40fd900 msgr2=0x7fa0b40ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.161+0000 7fa0ba80d700 1 -- 172.21.15.67:0/1076748536 shutdown_connections 2024-05-03T17:13:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.161+0000 7fa0ba80d700 1 -- 172.21.15.67:0/1076748536 wait complete. 2024-05-03T17:13:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.161+0000 7fa0ba80d700 1 Processor -- start 2024-05-03T17:13:32.163 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.161+0000 7fa0ba80d700 1 -- start start 2024-05-03T17:13:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.162+0000 7fa0ba80d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b41040b0 0x7fa0b419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.162+0000 7fa0ba80d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b410d430 con 0x7fa0b41040b0 2024-05-03T17:13:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.162+0000 7fa0b3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b41040b0 0x7fa0b419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.162+0000 7fa0b3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b41040b0 0x7fa0b419e600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55334/0 (socket says 172.21.15.67:55334) 2024-05-03T17:13:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.162+0000 7fa0b3fff700 1 -- 172.21.15.67:0/3531410483 learned_addr learned my addr 172.21.15.67:0/3531410483 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.162+0000 7fa0b3fff700 1 -- 172.21.15.67:0/3531410483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b419eb40 con 0x7fa0b41040b0 2024-05-03T17:13:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.162+0000 7fa0b3fff700 1 --2- 172.21.15.67:0/3531410483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b41040b0 0x7fa0b419e600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40056d0 tx=0x7fa0a40057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.163+0000 7fa0b17fa700 1 -- 172.21.15.67:0/3531410483 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0a4026070 con 0x7fa0b41040b0 2024-05-03T17:13:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.163+0000 7fa0b17fa700 1 -- 172.21.15.67:0/3531410483 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa0a4004370 con 0x7fa0b41040b0 2024-05-03T17:13:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.163+0000 7fa0ba80d700 1 -- 172.21.15.67:0/3531410483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b419edd0 con 0x7fa0b41040b0 2024-05-03T17:13:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.163+0000 7fa0b17fa700 1 -- 172.21.15.67:0/3531410483 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0a40039b0 con 0x7fa0b41040b0 2024-05-03T17:13:32.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.163+0000 7fa0ba80d700 1 -- 172.21.15.67:0/3531410483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b4198780 con 0x7fa0b41040b0 2024-05-03T17:13:32.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.164+0000 7fa0ba80d700 1 -- 172.21.15.67:0/3531410483 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b406bbf0 con 0x7fa0b41040b0 2024-05-03T17:13:32.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.164+0000 7fa0b17fa700 1 -- 172.21.15.67:0/3531410483 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fa0a4029070 con 0x7fa0b41040b0 2024-05-03T17:13:32.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.164+0000 7fa0b17fa700 1 --2- 172.21.15.67:0/3531410483 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa09c03e510 0x7fa09c0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:32.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.164+0000 7fa0b17fa700 1 -- 172.21.15.67:0/3531410483 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa0a401e070 con 0x7fa0b41040b0 2024-05-03T17:13:32.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.164+0000 7fa0b37fe700 1 --2- 172.21.15.67:0/3531410483 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa09c03e510 0x7fa09c0409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:32.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.165+0000 7fa0b37fe700 1 --2- 172.21.15.67:0/3531410483 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa09c03e510 0x7fa09c0409d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8006fd0 tx=0x7fa0a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:32.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.169+0000 7fa0b17fa700 1 -- 172.21.15.67:0/3531410483 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa0a4013070 con 0x7fa0b41040b0 2024-05-03T17:13:32.392 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.390+0000 7fa0ba80d700 1 -- 172.21.15.67:0/3531410483 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa0b4064b00 con 0x7fa09c03e510 2024-05-03T17:13:32.393 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.391+0000 7fa0b17fa700 1 -- 172.21.15.67:0/3531410483 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7fa0b4064b00 con 0x7fa09c03e510 2024-05-03T17:13:32.393 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:13:32.393 INFO:teuthology.orchestra.run.smithi067.stdout:[{"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}, {"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}] 2024-05-03T17:13:32.395 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.394+0000 7fa0ba80d700 1 -- 172.21.15.67:0/3531410483 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa09c03e510 msgr2=0x7fa09c0409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:32.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.394+0000 7fa0ba80d700 1 --2- 172.21.15.67:0/3531410483 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa09c03e510 0x7fa09c0409d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8006fd0 tx=0x7fa0a8006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:32.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.394+0000 7fa0ba80d700 1 -- 172.21.15.67:0/3531410483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b41040b0 msgr2=0x7fa0b419e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:32.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.394+0000 7fa0ba80d700 1 --2- 172.21.15.67:0/3531410483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b41040b0 0x7fa0b419e600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa0a40056d0 tx=0x7fa0a40057b0 comp rx=0 tx=0).stop 2024-05-03T17:13:32.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.394+0000 7fa0ba80d700 1 -- 172.21.15.67:0/3531410483 shutdown_connections 2024-05-03T17:13:32.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.394+0000 7fa0ba80d700 1 --2- 172.21.15.67:0/3531410483 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa09c03e510 0x7fa09c0409d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:32.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.394+0000 7fa0ba80d700 1 --2- 172.21.15.67:0/3531410483 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa0b41040b0 0x7fa0b419e600 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:32.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.394+0000 7fa0ba80d700 1 -- 172.21.15.67:0/3531410483 >> 172.21.15.67:0/3531410483 conn(0x7fa0b40fd900 msgr2=0x7fa0b4104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:32.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.394+0000 7fa0ba80d700 1 -- 172.21.15.67:0/3531410483 shutdown_connections 2024-05-03T17:13:32.396 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:32.394+0000 7fa0ba80d700 1 -- 172.21.15.67:0/3531410483 wait complete. 2024-05-03T17:13:33.008 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi082 2024-05-03T17:13:33.008 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T17:13:33.008 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/etc/ceph/ceph.conf 2024-05-03T17:13:33.028 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T17:13:33.028 DEBUG:teuthology.orchestra.run.smithi082:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T17:13:33.085 INFO:tasks.cephadm:Adding host smithi082 to orchestrator... 2024-05-03T17:13:33.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch host add smithi082 2024-05-03T17:13:33.334 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:13:33.548 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:33 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:33.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:33 smithi067 ceph-mon[29731]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:33.549 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:33 smithi067 ceph-mon[29731]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:13:34.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.863+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2982456373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e58102050 msgr2=0x7f9e58102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.863+0000 7f9e5cb8d700 1 --2- 172.21.15.67:0/2982456373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e58102050 0x7f9e58102430 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9e5000b0a0 tx=0x7f9e5000b3b0 comp rx=0 tx=0).stop 2024-05-03T17:13:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.863+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2982456373 shutdown_connections 2024-05-03T17:13:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.863+0000 7f9e5cb8d700 1 --2- 172.21.15.67:0/2982456373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e58102050 0x7f9e58102430 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.863+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2982456373 >> 172.21.15.67:0/2982456373 conn(0x7f9e58077900 msgr2=0x7f9e58077d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.863+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2982456373 shutdown_connections 2024-05-03T17:13:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.863+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2982456373 wait complete. 2024-05-03T17:13:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.864+0000 7f9e5cb8d700 1 Processor -- start 2024-05-03T17:13:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.864+0000 7f9e5cb8d700 1 -- start start 2024-05-03T17:13:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.864+0000 7f9e5cb8d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e58102050 0x7f9e58103aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.864+0000 7f9e5cb8d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5810c280 con 0x7f9e58102050 2024-05-03T17:13:34.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.864+0000 7f9e5659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e58102050 0x7f9e58103aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:34.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.864+0000 7f9e5659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e58102050 0x7f9e58103aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55348/0 (socket says 172.21.15.67:55348) 2024-05-03T17:13:34.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.864+0000 7f9e5659c700 1 -- 172.21.15.67:0/2034153230 learned_addr learned my addr 172.21.15.67:0/2034153230 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:34.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.865+0000 7f9e5659c700 1 -- 172.21.15.67:0/2034153230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e58105990 con 0x7f9e58102050 2024-05-03T17:13:34.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.865+0000 7f9e5659c700 1 --2- 172.21.15.67:0/2034153230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e58102050 0x7f9e58103aa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9e50006130 tx=0x7f9e500097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:34.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.865+0000 7f9e4f7fe700 1 -- 172.21.15.67:0/2034153230 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e50021020 con 0x7f9e58102050 2024-05-03T17:13:34.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.865+0000 7f9e4f7fe700 1 -- 172.21.15.67:0/2034153230 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9e500079b0 con 0x7f9e58102050 2024-05-03T17:13:34.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.865+0000 7f9e4f7fe700 1 -- 172.21.15.67:0/2034153230 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9e50017040 con 0x7f9e58102050 2024-05-03T17:13:34.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.865+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2034153230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e58104040 con 0x7f9e58102050 2024-05-03T17:13:34.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.865+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2034153230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e58104500 con 0x7f9e58102050 2024-05-03T17:13:34.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.866+0000 7f9e4f7fe700 1 -- 172.21.15.67:0/2034153230 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f9e50024020 con 0x7f9e58102050 2024-05-03T17:13:34.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.866+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2034153230 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e581034e0 con 0x7f9e58102050 2024-05-03T17:13:34.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.866+0000 7f9e4f7fe700 1 --2- 172.21.15.67:0/2034153230 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9e4403e510 0x7f9e440409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:34.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.866+0000 7f9e4f7fe700 1 -- 172.21.15.67:0/2034153230 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9e5000f050 con 0x7f9e58102050 2024-05-03T17:13:34.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.866+0000 7f9e55d9b700 1 --2- 172.21.15.67:0/2034153230 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9e4403e510 0x7f9e440409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:34.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.867+0000 7f9e55d9b700 1 --2- 172.21.15.67:0/2034153230 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9e4403e510 0x7f9e440409d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9e40006fd0 tx=0x7f9e40006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:34.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:34.871+0000 7f9e4f7fe700 1 -- 172.21.15.67:0/2034153230 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9e5002b020 con 0x7f9e58102050 2024-05-03T17:13:35.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:35.090+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2034153230 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}) -- 0x7f9e58104fd0 con 0x7f9e4403e510 2024-05-03T17:13:35.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:35 smithi067 ceph-mon[29731]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:35.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:35 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:35.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:35 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:35.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:35 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:35.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:35 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:35.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:35 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi070", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:13:35.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:35 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:13:35.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:35 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:13:36.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:36 smithi067 ceph-mon[29731]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:13:36.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:36 smithi067 ceph-mon[29731]: Updating smithi070:/etc/ceph/ceph.conf 2024-05-03T17:13:36.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:36 smithi067 ceph-mon[29731]: Updating smithi070:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:13:36.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:36 smithi067 ceph-mon[29731]: Deploying cephadm binary to smithi082 2024-05-03T17:13:36.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:36 smithi067 ceph-mon[29731]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-05-03T17:13:37.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:37 smithi067 ceph-mon[29731]: Updating smithi070:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-03T17:13:37.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:37 smithi067 ceph-mon[29731]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:39.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:39.307+0000 7f9e4f7fe700 1 -- 172.21.15.67:0/2034153230 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f9e58104fd0 con 0x7f9e4403e510 2024-05-03T17:13:39.308 INFO:teuthology.orchestra.run.smithi067.stdout:Added host 'smithi082' with addr '172.21.15.82' 2024-05-03T17:13:39.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:39.310+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2034153230 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9e4403e510 msgr2=0x7f9e440409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:39.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:39.310+0000 7f9e5cb8d700 1 --2- 172.21.15.67:0/2034153230 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9e4403e510 0x7f9e440409d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9e40006fd0 tx=0x7f9e40006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:39.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:39.310+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2034153230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e58102050 msgr2=0x7f9e58103aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:39.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:39.310+0000 7f9e5cb8d700 1 --2- 172.21.15.67:0/2034153230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e58102050 0x7f9e58103aa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f9e50006130 tx=0x7f9e500097d0 comp rx=0 tx=0).stop 2024-05-03T17:13:39.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:39.310+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2034153230 shutdown_connections 2024-05-03T17:13:39.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:39.310+0000 7f9e5cb8d700 1 --2- 172.21.15.67:0/2034153230 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9e4403e510 0x7f9e440409d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:39.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:39.310+0000 7f9e5cb8d700 1 --2- 172.21.15.67:0/2034153230 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9e58102050 0x7f9e58103aa0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:39.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:39.310+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2034153230 >> 172.21.15.67:0/2034153230 conn(0x7f9e58077900 msgr2=0x7f9e5806a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:39.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:39.310+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2034153230 shutdown_connections 2024-05-03T17:13:39.311 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:39.310+0000 7f9e5cb8d700 1 -- 172.21.15.67:0/2034153230 wait complete. 2024-05-03T17:13:39.899 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:39 smithi067 ceph-mon[29731]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:39.899 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:39.899 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:39.899 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:39.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:39.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:13:39.935 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch host ls --format=json 2024-05-03T17:13:40.182 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:13:41.172 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:40 smithi067 ceph-mon[29731]: Added host smithi082 2024-05-03T17:13:41.172 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:40 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:41.172 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:40 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:41.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.662+0000 7f82bd4be700 1 -- 172.21.15.67:0/1016074339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8103140 msgr2=0x7f82b8103520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:41.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.662+0000 7f82bd4be700 1 --2- 172.21.15.67:0/1016074339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8103140 0x7f82b8103520 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f82ac009910 tx=0x7f82ac009c20 comp rx=0 tx=0).stop 2024-05-03T17:13:41.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.663+0000 7f82bd4be700 1 -- 172.21.15.67:0/1016074339 shutdown_connections 2024-05-03T17:13:41.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.663+0000 7f82bd4be700 1 --2- 172.21.15.67:0/1016074339 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8103140 0x7f82b8103520 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:41.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.663+0000 7f82bd4be700 1 -- 172.21.15.67:0/1016074339 >> 172.21.15.67:0/1016074339 conn(0x7f82b80fe9d0 msgr2=0x7f82b8100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:41.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.663+0000 7f82bd4be700 1 -- 172.21.15.67:0/1016074339 shutdown_connections 2024-05-03T17:13:41.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.663+0000 7f82bd4be700 1 -- 172.21.15.67:0/1016074339 wait complete. 2024-05-03T17:13:41.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.663+0000 7f82bd4be700 1 Processor -- start 2024-05-03T17:13:41.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.663+0000 7f82bd4be700 1 -- start start 2024-05-03T17:13:41.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.664+0000 7f82bd4be700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8103140 0x7f82b80757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:41.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.664+0000 7f82bd4be700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f82b810d3c0 con 0x7f82b8103140 2024-05-03T17:13:41.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.664+0000 7f82b6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8103140 0x7f82b80757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:41.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.664+0000 7f82b6ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8103140 0x7f82b80757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60926/0 (socket says 172.21.15.67:60926) 2024-05-03T17:13:41.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.664+0000 7f82b6ffd700 1 -- 172.21.15.67:0/2061843264 learned_addr learned my addr 172.21.15.67:0/2061843264 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:41.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.664+0000 7f82b6ffd700 1 -- 172.21.15.67:0/2061843264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82b8077690 con 0x7f82b8103140 2024-05-03T17:13:41.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.664+0000 7f82b6ffd700 1 --2- 172.21.15.67:0/2061843264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8103140 0x7f82b80757a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f82ac0056d0 tx=0x7f82ac0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:41.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.664+0000 7f829ffff700 1 -- 172.21.15.67:0/2061843264 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82ac026070 con 0x7f82b8103140 2024-05-03T17:13:41.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.664+0000 7f82bd4be700 1 -- 172.21.15.67:0/2061843264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82b8075d40 con 0x7f82b8103140 2024-05-03T17:13:41.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.664+0000 7f829ffff700 1 -- 172.21.15.67:0/2061843264 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f82ac00bc00 con 0x7f82b8103140 2024-05-03T17:13:41.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.664+0000 7f829ffff700 1 -- 172.21.15.67:0/2061843264 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f82ac003920 con 0x7f82b8103140 2024-05-03T17:13:41.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.664+0000 7f82bd4be700 1 -- 172.21.15.67:0/2061843264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82b8076200 con 0x7f82b8103140 2024-05-03T17:13:41.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.665+0000 7f82bd4be700 1 -- 172.21.15.67:0/2061843264 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82b8104620 con 0x7f82b8103140 2024-05-03T17:13:41.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.665+0000 7f829ffff700 1 -- 172.21.15.67:0/2061843264 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f82ac029070 con 0x7f82b8103140 2024-05-03T17:13:41.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.666+0000 7f829ffff700 1 --2- 172.21.15.67:0/2061843264 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f82a003e510 0x7f82a00409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:41.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.666+0000 7f829ffff700 1 -- 172.21.15.67:0/2061843264 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f82ac01e070 con 0x7f82b8103140 2024-05-03T17:13:41.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.666+0000 7f82b67fc700 1 --2- 172.21.15.67:0/2061843264 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f82a003e510 0x7f82a00409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:41.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.666+0000 7f82b67fc700 1 --2- 172.21.15.67:0/2061843264 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f82a003e510 0x7f82a00409d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f82a8006fd0 tx=0x7f82a8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:41.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.671+0000 7f829ffff700 1 -- 172.21.15.67:0/2061843264 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f82ac013070 con 0x7f82b8103140 2024-05-03T17:13:41.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.890+0000 7f82bd4be700 1 -- 172.21.15.67:0/2061843264 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f82b8076cd0 con 0x7f82a003e510 2024-05-03T17:13:41.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.891+0000 7f829ffff700 1 -- 172.21.15.67:0/2061843264 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+238 (secure 0 0 0) 0x7f82b8076cd0 con 0x7f82a003e510 2024-05-03T17:13:41.891 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:13:41.891 INFO:teuthology.orchestra.run.smithi067.stdout:[{"addr": "172.21.15.67", "hostname": "smithi067", "labels": [], "status": ""}, {"addr": "172.21.15.70", "hostname": "smithi070", "labels": [], "status": ""}, {"addr": "172.21.15.82", "hostname": "smithi082", "labels": [], "status": ""}] 2024-05-03T17:13:41.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.893+0000 7f82bd4be700 1 -- 172.21.15.67:0/2061843264 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f82a003e510 msgr2=0x7f82a00409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:41.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.893+0000 7f82bd4be700 1 --2- 172.21.15.67:0/2061843264 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f82a003e510 0x7f82a00409d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f82a8006fd0 tx=0x7f82a8006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.893+0000 7f82bd4be700 1 -- 172.21.15.67:0/2061843264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8103140 msgr2=0x7f82b80757a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.893+0000 7f82bd4be700 1 --2- 172.21.15.67:0/2061843264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8103140 0x7f82b80757a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f82ac0056d0 tx=0x7f82ac0057b0 comp rx=0 tx=0).stop 2024-05-03T17:13:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.893+0000 7f82bd4be700 1 -- 172.21.15.67:0/2061843264 shutdown_connections 2024-05-03T17:13:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.893+0000 7f82bd4be700 1 --2- 172.21.15.67:0/2061843264 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f82a003e510 0x7f82a00409d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.893+0000 7f82bd4be700 1 --2- 172.21.15.67:0/2061843264 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f82b8103140 0x7f82b80757a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.893+0000 7f82bd4be700 1 -- 172.21.15.67:0/2061843264 >> 172.21.15.67:0/2061843264 conn(0x7f82b80fe9d0 msgr2=0x7f82b8104b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.893+0000 7f82bd4be700 1 -- 172.21.15.67:0/2061843264 shutdown_connections 2024-05-03T17:13:41.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:41.893+0000 7f82bd4be700 1 -- 172.21.15.67:0/2061843264 wait complete. 2024-05-03T17:13:42.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:41 smithi067 ceph-mon[29731]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:42.535 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-03T17:13:42.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd crush tunables default 2024-05-03T17:13:42.781 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:13:43.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:42 smithi067 ceph-mon[29731]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:13:43.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:42 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:44.137 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:43 smithi067 ceph-mon[29731]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:44.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.371+0000 7faa8da46700 1 -- 172.21.15.67:0/1408007879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa880758d0 msgr2=0x7faa88075cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:44.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.371+0000 7faa8da46700 1 --2- 172.21.15.67:0/1408007879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa880758d0 0x7faa88075cb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7faa78009910 tx=0x7faa78009c20 comp rx=0 tx=0).stop 2024-05-03T17:13:44.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.371+0000 7faa8da46700 1 -- 172.21.15.67:0/1408007879 shutdown_connections 2024-05-03T17:13:44.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.371+0000 7faa8da46700 1 --2- 172.21.15.67:0/1408007879 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa880758d0 0x7faa88075cb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:44.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.371+0000 7faa8da46700 1 -- 172.21.15.67:0/1408007879 >> 172.21.15.67:0/1408007879 conn(0x7faa880fe9e0 msgr2=0x7faa88100e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:44.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.371+0000 7faa8da46700 1 -- 172.21.15.67:0/1408007879 shutdown_connections 2024-05-03T17:13:44.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.371+0000 7faa8da46700 1 -- 172.21.15.67:0/1408007879 wait complete. 2024-05-03T17:13:44.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.371+0000 7faa8da46700 1 Processor -- start 2024-05-03T17:13:44.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.371+0000 7faa8da46700 1 -- start start 2024-05-03T17:13:44.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.372+0000 7faa8da46700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa880758d0 0x7faa881a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:44.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.372+0000 7faa8da46700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faa8810f610 con 0x7faa880758d0 2024-05-03T17:13:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.372+0000 7faa86ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa880758d0 0x7faa881a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.372+0000 7faa86ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa880758d0 0x7faa881a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:60946/0 (socket says 172.21.15.67:60946) 2024-05-03T17:13:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.372+0000 7faa86ffd700 1 -- 172.21.15.67:0/2715476884 learned_addr learned my addr 172.21.15.67:0/2715476884 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:13:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.372+0000 7faa86ffd700 1 -- 172.21.15.67:0/2715476884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa881a0dd0 con 0x7faa880758d0 2024-05-03T17:13:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.372+0000 7faa86ffd700 1 --2- 172.21.15.67:0/2715476884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa880758d0 0x7faa881a0890 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faa780056d0 tx=0x7faa780057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.372+0000 7faa8ca44700 1 -- 172.21.15.67:0/2715476884 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa78026070 con 0x7faa880758d0 2024-05-03T17:13:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.372+0000 7faa8ca44700 1 -- 172.21.15.67:0/2715476884 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa78004370 con 0x7faa880758d0 2024-05-03T17:13:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.372+0000 7faa8ca44700 1 -- 172.21.15.67:0/2715476884 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faa780039b0 con 0x7faa880758d0 2024-05-03T17:13:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.372+0000 7faa8da46700 1 -- 172.21.15.67:0/2715476884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa881a1060 con 0x7faa880758d0 2024-05-03T17:13:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.373+0000 7faa8da46700 1 -- 172.21.15.67:0/2715476884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa8819aaa0 con 0x7faa880758d0 2024-05-03T17:13:44.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.373+0000 7faa8ca44700 1 -- 172.21.15.67:0/2715476884 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7faa78029070 con 0x7faa880758d0 2024-05-03T17:13:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.373+0000 7faa8da46700 1 -- 172.21.15.67:0/2715476884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa88076f80 con 0x7faa880758d0 2024-05-03T17:13:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.374+0000 7faa8ca44700 1 --2- 172.21.15.67:0/2715476884 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7faa7003e570 0x7faa70040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.374+0000 7faa8ca44700 1 -- 172.21.15.67:0/2715476884 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faa7801e070 con 0x7faa880758d0 2024-05-03T17:13:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.374+0000 7faa867fc700 1 --2- 172.21.15.67:0/2715476884 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7faa7003e570 0x7faa70040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:44.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.374+0000 7faa867fc700 1 --2- 172.21.15.67:0/2715476884 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7faa7003e570 0x7faa70040a30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7faa7c006fd0 tx=0x7faa7c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:44.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.379+0000 7faa8ca44700 1 -- 172.21.15.67:0/2715476884 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7faa78013070 con 0x7faa880758d0 2024-05-03T17:13:44.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.598+0000 7faa8da46700 1 -- 172.21.15.67:0/2715476884 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7faa88064b00 con 0x7faa880758d0 2024-05-03T17:13:44.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.910+0000 7faa8ca44700 1 -- 172.21.15.67:0/2715476884 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7faa7800bcc0 con 0x7faa880758d0 2024-05-03T17:13:44.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.914+0000 7faa8da46700 1 -- 172.21.15.67:0/2715476884 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7faa7003e570 msgr2=0x7faa70040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:44.914 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.914+0000 7faa8da46700 1 --2- 172.21.15.67:0/2715476884 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7faa7003e570 0x7faa70040a30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7faa7c006fd0 tx=0x7faa7c006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.914+0000 7faa8da46700 1 -- 172.21.15.67:0/2715476884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa880758d0 msgr2=0x7faa881a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.914+0000 7faa8da46700 1 --2- 172.21.15.67:0/2715476884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa880758d0 0x7faa881a0890 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faa780056d0 tx=0x7faa780057b0 comp rx=0 tx=0).stop 2024-05-03T17:13:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.914+0000 7faa8da46700 1 -- 172.21.15.67:0/2715476884 shutdown_connections 2024-05-03T17:13:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.914+0000 7faa8da46700 1 --2- 172.21.15.67:0/2715476884 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7faa7003e570 0x7faa70040a30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.914+0000 7faa8da46700 1 --2- 172.21.15.67:0/2715476884 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa880758d0 0x7faa881a0890 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.914+0000 7faa8da46700 1 -- 172.21.15.67:0/2715476884 >> 172.21.15.67:0/2715476884 conn(0x7faa880fe9e0 msgr2=0x7faa88107150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.914+0000 7faa8da46700 1 -- 172.21.15.67:0/2715476884 shutdown_connections 2024-05-03T17:13:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:13:44.914+0000 7faa8da46700 1 -- 172.21.15.67:0/2715476884 wait complete. 2024-05-03T17:13:44.915 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2024-05-03T17:13:45.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:44 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2715476884' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-03T17:13:45.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:45.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:45.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:45.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:45.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:13:45.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:13:45.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:13:45.619 INFO:tasks.cephadm:Adding mon.a on smithi067 2024-05-03T17:13:45.619 INFO:tasks.cephadm:Adding mon.b on smithi070 2024-05-03T17:13:45.619 INFO:tasks.cephadm:Adding mon.c on smithi082 2024-05-03T17:13:45.619 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch apply mon '3;smithi067:172.21.15.67=a;smithi070:172.21.15.70=b;smithi082:172.21.15.82=c' 2024-05-03T17:13:45.850 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:13:46.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:45 smithi067 ceph-mon[29731]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:46.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:45 smithi067 ceph-mon[29731]: Updating smithi082:/etc/ceph/ceph.conf 2024-05-03T17:13:46.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:45 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2715476884' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-03T17:13:46.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:45 smithi067 ceph-mon[29731]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T17:13:46.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:45 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:46.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:45 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:46.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:45 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:47.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:46 smithi067 ceph-mon[29731]: Updating smithi082:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:13:47.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:46 smithi067 ceph-mon[29731]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-05-03T17:13:47.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:46 smithi067 ceph-mon[29731]: Updating smithi082:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-03T17:13:47.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.279+0000 7f0a83807700 1 -- 172.21.15.82:0/2896901165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7c102010 msgr2=0x7f0a7c1023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:47.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.279+0000 7f0a83807700 1 --2- 172.21.15.82:0/2896901165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7c102010 0x7f0a7c1023f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0a78009910 tx=0x7f0a78009c20 comp rx=0 tx=0).stop 2024-05-03T17:13:47.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.279+0000 7f0a83807700 1 -- 172.21.15.82:0/2896901165 shutdown_connections 2024-05-03T17:13:47.283 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.279+0000 7f0a83807700 1 --2- 172.21.15.82:0/2896901165 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7c102010 0x7f0a7c1023f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.279+0000 7f0a83807700 1 -- 172.21.15.82:0/2896901165 >> 172.21.15.82:0/2896901165 conn(0x7f0a7c0fd8c0 msgr2=0x7f0a7c0ffce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.279+0000 7f0a83807700 1 -- 172.21.15.82:0/2896901165 shutdown_connections 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.279+0000 7f0a83807700 1 -- 172.21.15.82:0/2896901165 wait complete. 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.280+0000 7f0a83807700 1 Processor -- start 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.280+0000 7f0a83807700 1 -- start start 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.280+0000 7f0a83807700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7c102010 0x7f0a7c197540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.280+0000 7f0a83807700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7c10c290 con 0x7f0a7c102010 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.280+0000 7f0a815a3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7c102010 0x7f0a7c197540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.280+0000 7f0a815a3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7c102010 0x7f0a7c197540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.82:56708/0 (socket says 172.21.15.82:56708) 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.280+0000 7f0a815a3700 1 -- 172.21.15.82:0/171215120 learned_addr learned my addr 172.21.15.82:0/171215120 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.281+0000 7f0a815a3700 1 -- 172.21.15.82:0/171215120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a7c06c130 con 0x7f0a7c102010 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.281+0000 7f0a815a3700 1 --2- 172.21.15.82:0/171215120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7c102010 0x7f0a7c197540 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0a78006010 tx=0x7f0a78005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.281+0000 7f0a727fc700 1 -- 172.21.15.82:0/171215120 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a78026070 con 0x7f0a7c102010 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.281+0000 7f0a83807700 1 -- 172.21.15.82:0/171215120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a7c197a80 con 0x7f0a7c102010 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.281+0000 7f0a727fc700 1 -- 172.21.15.82:0/171215120 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a7800bba0 con 0x7f0a7c102010 2024-05-03T17:13:47.284 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.281+0000 7f0a83807700 1 -- 172.21.15.82:0/171215120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a7c197f40 con 0x7f0a7c102010 2024-05-03T17:13:47.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.281+0000 7f0a727fc700 1 -- 172.21.15.82:0/171215120 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a780038a0 con 0x7f0a7c102010 2024-05-03T17:13:47.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.282+0000 7f0a727fc700 1 -- 172.21.15.82:0/171215120 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f0a78029070 con 0x7f0a7c102010 2024-05-03T17:13:47.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.282+0000 7f0a727fc700 1 --2- 172.21.15.82:0/171215120 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0a6803e510 0x7f0a680409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:47.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.282+0000 7f0a83807700 1 -- 172.21.15.82:0/171215120 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a7c1034f0 con 0x7f0a7c102010 2024-05-03T17:13:47.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.282+0000 7f0a727fc700 1 -- 172.21.15.82:0/171215120 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0a7801e070 con 0x7f0a7c102010 2024-05-03T17:13:47.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.283+0000 7f0a80da2700 1 --2- 172.21.15.82:0/171215120 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0a6803e510 0x7f0a680409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:47.285 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.283+0000 7f0a80da2700 1 --2- 172.21.15.82:0/171215120 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0a6803e510 0x7f0a680409d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c006fd0 tx=0x7f0a6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:47.289 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.288+0000 7f0a727fc700 1 -- 172.21.15.82:0/171215120 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f0a78013070 con 0x7f0a7c102010 2024-05-03T17:13:47.503 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.503+0000 7f0a83807700 1 -- 172.21.15.82:0/171215120 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi067:172.21.15.67=a;smithi070:172.21.15.70=b;smithi082:172.21.15.82=c", "target": ["mon-mgr", ""]}) -- 0x7f0a7c06b990 con 0x7f0a6803e510 2024-05-03T17:13:47.510 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.509+0000 7f0a727fc700 1 -- 172.21.15.82:0/171215120 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f0a7c06b990 con 0x7f0a6803e510 2024-05-03T17:13:47.510 INFO:teuthology.orchestra.run.smithi082.stdout:Scheduled mon update... 2024-05-03T17:13:47.512 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.511+0000 7f0a83807700 1 -- 172.21.15.82:0/171215120 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0a6803e510 msgr2=0x7f0a680409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:47.512 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.511+0000 7f0a83807700 1 --2- 172.21.15.82:0/171215120 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0a6803e510 0x7f0a680409d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c006fd0 tx=0x7f0a6c006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:47.512 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.511+0000 7f0a83807700 1 -- 172.21.15.82:0/171215120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7c102010 msgr2=0x7f0a7c197540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:47.512 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.511+0000 7f0a83807700 1 --2- 172.21.15.82:0/171215120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7c102010 0x7f0a7c197540 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0a78006010 tx=0x7f0a78005780 comp rx=0 tx=0).stop 2024-05-03T17:13:47.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.512+0000 7f0a83807700 1 -- 172.21.15.82:0/171215120 shutdown_connections 2024-05-03T17:13:47.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.512+0000 7f0a83807700 1 --2- 172.21.15.82:0/171215120 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0a6803e510 0x7f0a680409d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:47.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.512+0000 7f0a83807700 1 --2- 172.21.15.82:0/171215120 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a7c102010 0x7f0a7c197540 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:47.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.512+0000 7f0a83807700 1 -- 172.21.15.82:0/171215120 >> 172.21.15.82:0/171215120 conn(0x7f0a7c0fd8c0 msgr2=0x7f0a7c1039e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:47.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.512+0000 7f0a83807700 1 -- 172.21.15.82:0/171215120 shutdown_connections 2024-05-03T17:13:47.513 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:47.512+0000 7f0a83807700 1 -- 172.21.15.82:0/171215120 wait complete. 2024-05-03T17:13:48.101 DEBUG:teuthology.orchestra.run.smithi070:mon.b> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.b.service 2024-05-03T17:13:48.104 DEBUG:teuthology.orchestra.run.smithi082:mon.c> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.c.service 2024-05-03T17:13:48.106 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-03T17:13:48.106 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-03T17:13:48.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:47 smithi067 ceph-mon[29731]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:48.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:48.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:13:48.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:13:48.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:13:48.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:48.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:13:48.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:13:49.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:48 smithi067 ceph-mon[29731]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi067:172.21.15.67=a;smithi070:172.21.15.70=b;smithi082:172.21.15.82=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:13:49.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:48 smithi067 ceph-mon[29731]: Saving service mon spec with placement smithi067:172.21.15.67=a;smithi070:172.21.15.70=b;smithi082:172.21.15.82=c;count:3 2024-05-03T17:13:49.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:48 smithi067 ceph-mon[29731]: Deploying daemon mon.c on smithi082 2024-05-03T17:13:49.290 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:13:50.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:49 smithi067 ceph-mon[29731]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:51.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.796+0000 7f702ef11700 1 -- 172.21.15.82:0/3851044834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7028102050 msgr2=0x7f7028102430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:51.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.796+0000 7f702ef11700 1 --2- 172.21.15.82:0/3851044834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7028102050 0x7f7028102430 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f701c009910 tx=0x7f701c009c20 comp rx=0 tx=0).stop 2024-05-03T17:13:51.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.796+0000 7f702ef11700 1 -- 172.21.15.82:0/3851044834 shutdown_connections 2024-05-03T17:13:51.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.796+0000 7f702ef11700 1 --2- 172.21.15.82:0/3851044834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7028102050 0x7f7028102430 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:51.799 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.796+0000 7f702ef11700 1 -- 172.21.15.82:0/3851044834 >> 172.21.15.82:0/3851044834 conn(0x7f7028077900 msgr2=0x7f7028077d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.796+0000 7f702ef11700 1 -- 172.21.15.82:0/3851044834 shutdown_connections 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.796+0000 7f702ef11700 1 -- 172.21.15.82:0/3851044834 wait complete. 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.797+0000 7f702ef11700 1 Processor -- start 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.797+0000 7f702ef11700 1 -- start start 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.797+0000 7f702ef11700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7028102050 0x7f702819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.797+0000 7f702ef11700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f702810c280 con 0x7f7028102050 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.797+0000 7f702ccad700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7028102050 0x7f702819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.797+0000 7f702ccad700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7028102050 0x7f702819d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.82:54820/0 (socket says 172.21.15.82:54820) 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.797+0000 7f702ccad700 1 -- 172.21.15.82:0/3204911244 learned_addr learned my addr 172.21.15.82:0/3204911244 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.798+0000 7f702ccad700 1 -- 172.21.15.82:0/3204911244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f702819d990 con 0x7f7028102050 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.798+0000 7f702ccad700 1 --2- 172.21.15.82:0/3204911244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7028102050 0x7f702819d450 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f701c0056d0 tx=0x7f701c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.798+0000 7f7025ffb700 1 -- 172.21.15.82:0/3204911244 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f701c026070 con 0x7f7028102050 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.798+0000 7f7025ffb700 1 -- 172.21.15.82:0/3204911244 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f701c00bba0 con 0x7f7028102050 2024-05-03T17:13:51.800 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.798+0000 7f702ef11700 1 -- 172.21.15.82:0/3204911244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f702819dc80 con 0x7f7028102050 2024-05-03T17:13:51.801 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.798+0000 7f7025ffb700 1 -- 172.21.15.82:0/3204911244 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f701c003920 con 0x7f7028102050 2024-05-03T17:13:51.801 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.798+0000 7f702ef11700 1 -- 172.21.15.82:0/3204911244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7028197610 con 0x7f7028102050 2024-05-03T17:13:51.802 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.799+0000 7f702ef11700 1 -- 172.21.15.82:0/3204911244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70281034e0 con 0x7f7028102050 2024-05-03T17:13:51.802 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.799+0000 7f7025ffb700 1 -- 172.21.15.82:0/3204911244 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f701c029070 con 0x7f7028102050 2024-05-03T17:13:51.802 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.799+0000 7f7025ffb700 1 --2- 172.21.15.82:0/3204911244 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f701803e510 0x7f70180409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:51.802 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.799+0000 7f7025ffb700 1 -- 172.21.15.82:0/3204911244 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f701c01e070 con 0x7f7028102050 2024-05-03T17:13:51.802 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.800+0000 7f7027fff700 1 --2- 172.21.15.82:0/3204911244 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f701803e510 0x7f70180409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:51.802 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.800+0000 7f7027fff700 1 --2- 172.21.15.82:0/3204911244 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f701803e510 0x7f70180409d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7014006fd0 tx=0x7f7014006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:51.805 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:51.805+0000 7f7025ffb700 1 -- 172.21.15.82:0/3204911244 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f701c013070 con 0x7f7028102050 2024-05-03T17:13:52.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:52.116+0000 7f702ef11700 1 -- 172.21.15.82:0/3204911244 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7028064b00 con 0x7f7028102050 2024-05-03T17:13:52.117 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:52.116+0000 7f7025ffb700 1 -- 172.21.15.82:0/3204911244 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f701c030090 con 0x7f7028102050 2024-05-03T17:13:52.117 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:13:52.118 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":1,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","modified":"2024-05-03T17:12:08.010048Z","created":"2024-05-03T17:12:08.010048Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-03T17:13:52.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:52.119+0000 7f702ef11700 1 -- 172.21.15.82:0/3204911244 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f701803e510 msgr2=0x7f70180409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:52.119 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:52.119+0000 7f702ef11700 1 --2- 172.21.15.82:0/3204911244 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f701803e510 0x7f70180409d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7014006fd0 tx=0x7f7014006e40 comp rx=0 tx=0).stop 2024-05-03T17:13:52.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:52.119+0000 7f702ef11700 1 -- 172.21.15.82:0/3204911244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7028102050 msgr2=0x7f702819d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:52.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:52.119+0000 7f702ef11700 1 --2- 172.21.15.82:0/3204911244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7028102050 0x7f702819d450 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f701c0056d0 tx=0x7f701c0057b0 comp rx=0 tx=0).stop 2024-05-03T17:13:52.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:52.119+0000 7f702ef11700 1 -- 172.21.15.82:0/3204911244 shutdown_connections 2024-05-03T17:13:52.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:52.119+0000 7f702ef11700 1 --2- 172.21.15.82:0/3204911244 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f701803e510 0x7f70180409d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:52.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:52.119+0000 7f702ef11700 1 --2- 172.21.15.82:0/3204911244 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7028102050 0x7f702819d450 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:52.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:52.119+0000 7f702ef11700 1 -- 172.21.15.82:0/3204911244 >> 172.21.15.82:0/3204911244 conn(0x7f7028077900 msgr2=0x7f7028109b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:52.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:52.119+0000 7f702ef11700 1 -- 172.21.15.82:0/3204911244 shutdown_connections 2024-05-03T17:13:52.120 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:52.119+0000 7f702ef11700 1 -- 172.21.15.82:0/3204911244 wait complete. 2024-05-03T17:13:52.120 INFO:teuthology.orchestra.run.smithi082.stderr:dumped monmap epoch 1 2024-05-03T17:13:52.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:51 smithi067 ceph-mon[29731]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:53.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:52 smithi067 ceph-mon[29731]: from='client.? 172.21.15.82:0/3204911244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T17:13:53.238 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-03T17:13:53.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-03T17:13:53.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi070", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi082", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: Updating smithi070:/etc/ceph/ceph.conf 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: Updating smithi070:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: Deploying cephadm binary to smithi082 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: Updating smithi070:/etc/ceph/ceph.client.admin.keyring 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: Updating smithi070:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:53.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: Added host smithi082 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2715476884' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-03T17:13:53.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi082", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: Updating smithi082:/etc/ceph/ceph.conf 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2715476884' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: Updating smithi082:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: Updating smithi082:/etc/ceph/ceph.client.admin.keyring 2024-05-03T17:13:53.519 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: Updating smithi082:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.client.admin.keyring 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi067:172.21.15.67=a;smithi070:172.21.15.70=b;smithi082:172.21.15.82=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: Saving service mon spec with placement smithi067:172.21.15.67=a;smithi070:172.21.15.70=b;smithi082:172.21.15.82=c;count:3 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: Deploying daemon mon.c on smithi082 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: from='client.? 172.21.15.82:0/3204911244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T17:13:53.520 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 ceph-mon[33535]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-03T17:13:53.521 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 bash[33515]: 2c1215b5a58674e908f6e47886ac0e96c3d81e5c0818f32b3f445e1d1a87684b 2024-05-03T17:13:53.521 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:53 smithi082 systemd[1]: Started Ceph mon.c for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:13:53.752 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-03T17:13:53.752 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-03T17:13:53.986 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:13:58.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.437+0000 7f592a2e8700 1 -- 172.21.15.82:0/3990828685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f590c0035c0 msgr2=0x7f590c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:58.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.437+0000 7f592a2e8700 1 --2- 172.21.15.82:0/3990828685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f590c0035c0 0x7f590c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5924102ec0 tx=0x7f591400feb0 comp rx=0 tx=0).stop 2024-05-03T17:13:58.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.437+0000 7f592a2e8700 1 -- 172.21.15.82:0/3990828685 shutdown_connections 2024-05-03T17:13:58.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.437+0000 7f592a2e8700 1 --2- 172.21.15.82:0/3990828685 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f590c0035c0 0x7f590c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:58.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.437+0000 7f592a2e8700 1 -- 172.21.15.82:0/3990828685 >> 172.21.15.82:0/3990828685 conn(0x7f59240feaa0 msgr2=0x7f5924100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:58.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.437+0000 7f592a2e8700 1 -- 172.21.15.82:0/3990828685 shutdown_connections 2024-05-03T17:13:58.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.437+0000 7f592a2e8700 1 -- 172.21.15.82:0/3990828685 wait complete. 2024-05-03T17:13:58.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.438+0000 7f592a2e8700 1 Processor -- start 2024-05-03T17:13:58.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.438+0000 7f592a2e8700 1 -- start start 2024-05-03T17:13:58.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.439+0000 7f592a2e8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f590c0035c0 0x7f59241a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:58.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.439+0000 7f592a2e8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59241a0da0 0x7f592419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:58.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.439+0000 7f592a2e8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f592410aee0 con 0x7f59241a0da0 2024-05-03T17:13:58.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.439+0000 7f592a2e8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f592410ad60 con 0x7f590c0035c0 2024-05-03T17:13:58.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.439+0000 7f5923fff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f590c0035c0 0x7f59241a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:58.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.439+0000 7f5923fff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f590c0035c0 0x7f59241a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47440/0 (socket says 172.21.15.82:47440) 2024-05-03T17:13:58.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.439+0000 7f5923fff700 1 -- 172.21.15.82:0/2764545687 learned_addr learned my addr 172.21.15.82:0/2764545687 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T17:13:58.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.439+0000 7f5923fff700 1 -- 172.21.15.82:0/2764545687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f590c0035c0 msgr2=0x7f59241a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-05-03T17:13:58.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.439+0000 7f5923fff700 1 -- 172.21.15.82:0/2764545687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f590c0035c0 msgr2=0x7f59241a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-03T17:13:58.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.439+0000 7f5923fff700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f590c0035c0 0x7f59241a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-03T17:13:58.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.439+0000 7f59237fe700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59241a0da0 0x7f592419aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:58.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.439+0000 7f5923fff700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f590c0035c0 0x7f59241a0860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-03T17:13:58.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.440+0000 7f59237fe700 1 -- 172.21.15.82:0/2764545687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f590c0035c0 msgr2=0x7f59241a0860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-03T17:13:58.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.440+0000 7f59237fe700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f590c0035c0 0x7f59241a0860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:58.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.440+0000 7f59237fe700 1 -- 172.21.15.82:0/2764545687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59241a13e0 con 0x7f59241a0da0 2024-05-03T17:13:58.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.440+0000 7f59237fe700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59241a0da0 0x7f592419aa20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f591800b4d0 tx=0x7f591800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:58.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.440+0000 7f59217fa700 1 -- 172.21.15.82:0/2764545687 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f5918011630 con 0x7f59241a0da0 2024-05-03T17:13:58.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.440+0000 7f592a2e8700 1 -- 172.21.15.82:0/2764545687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f592419af60 con 0x7f59241a0da0 2024-05-03T17:13:58.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.440+0000 7f59217fa700 1 -- 172.21.15.82:0/2764545687 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5918011c80 con 0x7f59241a0da0 2024-05-03T17:13:58.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.440+0000 7f59217fa700 1 -- 172.21.15.82:0/2764545687 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f5918010da0 con 0x7f59241a0da0 2024-05-03T17:13:58.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.440+0000 7f592a2e8700 1 -- 172.21.15.82:0/2764545687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f592419b4f0 con 0x7f59241a0da0 2024-05-03T17:13:58.442 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.441+0000 7f59217fa700 1 -- 172.21.15.82:0/2764545687 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f5918018020 con 0x7f59241a0da0 2024-05-03T17:13:58.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.441+0000 7f592a2e8700 1 -- 172.21.15.82:0/2764545687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59241046d0 con 0x7f59241a0da0 2024-05-03T17:13:58.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.441+0000 7f59217fa700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f591003e2b0 0x7f5910040770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:13:58.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.441+0000 7f59217fa700 1 -- 172.21.15.82:0/2764545687 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5918052180 con 0x7f59241a0da0 2024-05-03T17:13:58.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.442+0000 7f5923fff700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f591003e2b0 0x7f5910040770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:13:58.443 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.442+0000 7f5923fff700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f591003e2b0 0x7f5910040770 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f591400fa60 tx=0x7f5914005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:13:58.448 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.447+0000 7f59217fa700 1 -- 172.21.15.82:0/2764545687 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f591801f020 con 0x7f59241a0da0 2024-05-03T17:13:58.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: mon.a calling monitor election 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: mon.c calling monitor election 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: monmap epoch 2 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: fsid 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:13:58.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: last_changed 2024-05-03T17:13:53.244215+0000 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: created 2024-05-03T17:12:08.010048+0000 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: min_mon_release 19 (squid) 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: election_strategy: 1 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: 1: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.c 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: fsmap 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: mgrmap e13: x(active, since 51s) 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: overall HEALTH_OK 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:13:58.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:13:58.736 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.735+0000 7f592a2e8700 1 -- 172.21.15.82:0/2764545687 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f592419bf50 con 0x7f59241a0da0 2024-05-03T17:13:58.736 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:13:58.736 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: mon.a calling monitor election 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: mon.c calling monitor election 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: monmap epoch 2 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: fsid 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: last_changed 2024-05-03T17:13:53.244215+0000 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: created 2024-05-03T17:12:08.010048+0000 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: min_mon_release 19 (squid) 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: election_strategy: 1 2024-05-03T17:13:58.737 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-03T17:13:58.738 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: 1: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.c 2024-05-03T17:13:58.738 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: fsmap 2024-05-03T17:13:58.738 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T17:13:58.738 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: mgrmap e13: x(active, since 51s) 2024-05-03T17:13:58.738 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: overall HEALTH_OK 2024-05-03T17:13:58.738 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:58.738 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:58.738 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:13:58.738 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:13:58.738 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:13:58.738 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.736+0000 7f59217fa700 1 -- 172.21.15.82:0/2764545687 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7f59180187b0 con 0x7f59241a0da0 2024-05-03T17:13:58.739 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:13:58.739 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":2,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","modified":"2024-05-03T17:13:53.244215Z","created":"2024-05-03T17:12:08.010048Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-03T17:13:58.739 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.738+0000 7f592a2e8700 1 -- 172.21.15.82:0/2764545687 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f591003e2b0 msgr2=0x7f5910040770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:58.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.738+0000 7f592a2e8700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f591003e2b0 0x7f5910040770 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f591400fa60 tx=0x7f5914005c20 comp rx=0 tx=0).stop 2024-05-03T17:13:58.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.738+0000 7f592a2e8700 1 -- 172.21.15.82:0/2764545687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59241a0da0 msgr2=0x7f592419aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:13:58.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.738+0000 7f592a2e8700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59241a0da0 0x7f592419aa20 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f591800b4d0 tx=0x7f591800b890 comp rx=0 tx=0).stop 2024-05-03T17:13:58.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.738+0000 7f592a2e8700 1 -- 172.21.15.82:0/2764545687 shutdown_connections 2024-05-03T17:13:58.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.738+0000 7f592a2e8700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f591003e2b0 0x7f5910040770 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:58.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.738+0000 7f592a2e8700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f590c0035c0 0x7f59241a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:58.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.738+0000 7f592a2e8700 1 --2- 172.21.15.82:0/2764545687 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f59241a0da0 0x7f592419aa20 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:13:58.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.738+0000 7f592a2e8700 1 -- 172.21.15.82:0/2764545687 >> 172.21.15.82:0/2764545687 conn(0x7f59240feaa0 msgr2=0x7f5924104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:13:58.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.738+0000 7f592a2e8700 1 -- 172.21.15.82:0/2764545687 shutdown_connections 2024-05-03T17:13:58.740 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:13:58.738+0000 7f592a2e8700 1 -- 172.21.15.82:0/2764545687 wait complete. 2024-05-03T17:13:58.740 INFO:teuthology.orchestra.run.smithi082.stderr:dumped monmap epoch 2 2024-05-03T17:13:59.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:59 smithi067 ceph-mon[29731]: Deploying daemon mon.b on smithi070 2024-05-03T17:13:59.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:59 smithi067 ceph-mon[29731]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:59.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:59 smithi067 ceph-mon[29731]: from='client.? 172.21.15.82:0/2764545687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T17:13:59.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:13:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:13:59.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:59 smithi082 ceph-mon[33535]: Deploying daemon mon.b on smithi070 2024-05-03T17:13:59.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:59 smithi082 ceph-mon[33535]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:13:59.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:59 smithi082 ceph-mon[33535]: from='client.? 172.21.15.82:0/2764545687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T17:13:59.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:13:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:14:00.281 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-03T17:14:00.282 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-03T17:14:00.515 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:14:01.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:01 smithi067 ceph-mon[29731]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:01.730 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:01 smithi082 ceph-mon[33535]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:01.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.830+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/1100733239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f0109060 msgr2=0x7ff0f0109440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:01.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.830+0000 7ff0f7fd7700 1 --2- 172.21.15.82:0/1100733239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f0109060 0x7ff0f0109440 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009910 tx=0x7ff0ec009c20 comp rx=0 tx=0).stop 2024-05-03T17:14:01.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.830+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/1100733239 shutdown_connections 2024-05-03T17:14:01.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.830+0000 7ff0f7fd7700 1 --2- 172.21.15.82:0/1100733239 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f0109060 0x7ff0f0109440 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:01.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.830+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/1100733239 >> 172.21.15.82:0/1100733239 conn(0x7ff0f0077200 msgr2=0x7ff0f0077610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:14:01.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.831+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/1100733239 shutdown_connections 2024-05-03T17:14:01.839 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.831+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/1100733239 wait complete. 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.831+0000 7ff0f7fd7700 1 Processor -- start 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.831+0000 7ff0f7fd7700 1 -- start start 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.831+0000 7ff0f7fd7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f0109060 0x7ff0f019e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.831+0000 7ff0f7fd7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0f019ed10 0x7ff0f0198990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.831+0000 7ff0f7fd7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f010d4d0 con 0x7ff0f0109060 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.831+0000 7ff0f7fd7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f010d350 con 0x7ff0f019ed10 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5d73700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f0109060 0x7ff0f019e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5572700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0f019ed10 0x7ff0f0198990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5d73700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f0109060 0x7ff0f019e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.82:36810/0 (socket says 172.21.15.82:36810) 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5572700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0f019ed10 0x7ff0f0198990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:47456/0 (socket says 172.21.15.82:47456) 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5d73700 1 -- 172.21.15.82:0/2564453839 learned_addr learned my addr 172.21.15.82:0/2564453839 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5572700 1 -- 172.21.15.82:0/2564453839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0f019ed10 msgr2=0x7ff0f0198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5572700 1 -- 172.21.15.82:0/2564453839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0f019ed10 msgr2=0x7ff0f0198990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5572700 1 --2- 172.21.15.82:0/2564453839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0f019ed10 0x7ff0f0198990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-03T17:14:01.840 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5572700 1 --2- 172.21.15.82:0/2564453839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0f019ed10 0x7ff0f0198990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5d73700 1 -- 172.21.15.82:0/2564453839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0f019ed10 msgr2=0x7ff0f0198990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5d73700 1 --2- 172.21.15.82:0/2564453839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0f019ed10 0x7ff0f0198990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5d73700 1 -- 172.21.15.82:0/2564453839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f019f350 con 0x7ff0f0109060 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f5d73700 1 --2- 172.21.15.82:0/2564453839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f0109060 0x7ff0f019e7d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec006010 tx=0x7ff0ec005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0e6ffd700 1 -- 172.21.15.82:0/2564453839 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ff0ec005dc0 con 0x7ff0f0109060 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/2564453839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f0198ed0 con 0x7ff0f0109060 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0e6ffd700 1 -- 172.21.15.82:0/2564453839 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0ec00be10 con 0x7ff0f0109060 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0e6ffd700 1 -- 172.21.15.82:0/2564453839 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ff0ec022e50 con 0x7ff0f0109060 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.832+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/2564453839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f01994e0 con 0x7ff0f0109060 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.833+0000 7ff0e6ffd700 1 -- 172.21.15.82:0/2564453839 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7ff0ec02a070 con 0x7ff0f0109060 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.833+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/2564453839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0f0199790 con 0x7ff0f0109060 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.834+0000 7ff0e6ffd700 1 --2- 172.21.15.82:0/2564453839 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff0dc03e450 0x7ff0dc040910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.834+0000 7ff0e6ffd700 1 -- 172.21.15.82:0/2564453839 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff0ec01e030 con 0x7ff0f0109060 2024-05-03T17:14:01.841 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.834+0000 7ff0f5572700 1 --2- 172.21.15.82:0/2564453839 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff0dc03e450 0x7ff0dc040910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:01.842 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.834+0000 7ff0f5572700 1 --2- 172.21.15.82:0/2564453839 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff0dc03e450 0x7ff0dc040910 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009850 tx=0x7ff0e0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:14:01.842 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:01.839+0000 7ff0e6ffd700 1 -- 172.21.15.82:0/2564453839 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ff0ec013070 con 0x7ff0f0109060 2024-05-03T17:14:02.127 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.126+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/2564453839 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff0f0199ec0 con 0x7ff0f0109060 2024-05-03T17:14:02.127 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.127+0000 7ff0e6ffd700 1 -- 172.21.15.82:0/2564453839 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1011 (secure 0 0 0) 0x7ff0ec00d340 con 0x7ff0f0109060 2024-05-03T17:14:02.127 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:14:02.128 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":2,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","modified":"2024-05-03T17:13:53.244215Z","created":"2024-05-03T17:12:08.010048Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-03T17:14:02.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.129+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/2564453839 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff0dc03e450 msgr2=0x7ff0dc040910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:02.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.129+0000 7ff0f7fd7700 1 --2- 172.21.15.82:0/2564453839 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff0dc03e450 0x7ff0dc040910 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009850 tx=0x7ff0e0008040 comp rx=0 tx=0).stop 2024-05-03T17:14:02.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.129+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/2564453839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f0109060 msgr2=0x7ff0f019e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:02.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.129+0000 7ff0f7fd7700 1 --2- 172.21.15.82:0/2564453839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f0109060 0x7ff0f019e7d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec006010 tx=0x7ff0ec005030 comp rx=0 tx=0).stop 2024-05-03T17:14:02.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.129+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/2564453839 shutdown_connections 2024-05-03T17:14:02.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.129+0000 7ff0f7fd7700 1 --2- 172.21.15.82:0/2564453839 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff0dc03e450 0x7ff0dc040910 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:02.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.129+0000 7ff0f7fd7700 1 --2- 172.21.15.82:0/2564453839 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff0f0109060 0x7ff0f019e7d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:02.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.129+0000 7ff0f7fd7700 1 --2- 172.21.15.82:0/2564453839 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0f019ed10 0x7ff0f0198990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:02.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.129+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/2564453839 >> 172.21.15.82:0/2564453839 conn(0x7ff0f0077200 msgr2=0x7ff0f00fed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:14:02.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.129+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/2564453839 shutdown_connections 2024-05-03T17:14:02.130 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:02.129+0000 7ff0f7fd7700 1 -- 172.21.15.82:0/2564453839 wait complete. 2024-05-03T17:14:02.131 INFO:teuthology.orchestra.run.smithi082.stderr:dumped monmap epoch 2 2024-05-03T17:14:02.524 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:02 smithi070 systemd[1]: Started Ceph mon.b for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:14:02.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:02 smithi067 ceph-mon[29731]: from='client.? 172.21.15.82:0/2564453839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T17:14:02.718 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:02 smithi082 ceph-mon[33535]: from='client.? 172.21.15.82:0/2564453839' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T17:14:03.745 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-03T17:14:03.745 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph mon dump -f json 2024-05-03T17:14:03.987 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:14:07.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:07.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:14:07.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: mon.a calling monitor election 2024-05-03T17:14:07.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:14:07.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: mon.c calling monitor election 2024-05-03T17:14:07.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: mon.b calling monitor election 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: monmap epoch 3 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: fsid 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: last_changed 2024-05-03T17:14:02.406776+0000 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: created 2024-05-03T17:12:08.010048+0000 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: min_mon_release 19 (squid) 2024-05-03T17:14:07.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: election_strategy: 1 2024-05-03T17:14:07.767 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-03T17:14:07.767 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: 1: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.c 2024-05-03T17:14:07.767 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: 2: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.b 2024-05-03T17:14:07.767 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: fsmap 2024-05-03T17:14:07.767 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T17:14:07.767 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: mgrmap e13: x(active, since 61s) 2024-05-03T17:14:07.767 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: overall HEALTH_OK 2024-05-03T17:14:07.767 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:07.767 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:07.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:07.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:14:07.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: mon.a calling monitor election 2024-05-03T17:14:07.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:14:07.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: mon.c calling monitor election 2024-05-03T17:14:07.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:07.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: mon.b calling monitor election 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: monmap epoch 3 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: fsid 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: last_changed 2024-05-03T17:14:02.406776+0000 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: created 2024-05-03T17:12:08.010048+0000 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: min_mon_release 19 (squid) 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: election_strategy: 1 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: 1: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.c 2024-05-03T17:14:07.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: 2: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.b 2024-05-03T17:14:07.811 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: fsmap 2024-05-03T17:14:07.811 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T17:14:07.811 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: mgrmap e13: x(active, since 61s) 2024-05-03T17:14:07.811 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: overall HEALTH_OK 2024-05-03T17:14:07.811 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:07.811 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:07.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:07.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:14:07.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: mon.a calling monitor election 2024-05-03T17:14:07.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:14:07.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: mon.c calling monitor election 2024-05-03T17:14:07.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:07.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: mon.b calling monitor election 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: monmap epoch 3 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: fsid 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: last_changed 2024-05-03T17:14:02.406776+0000 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: created 2024-05-03T17:12:08.010048+0000 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: min_mon_release 19 (squid) 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: election_strategy: 1 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: 0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: 1: [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] mon.c 2024-05-03T17:14:07.955 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: 2: [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] mon.b 2024-05-03T17:14:07.956 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: fsmap 2024-05-03T17:14:07.956 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: osdmap e4: 0 total, 0 up, 0 in 2024-05-03T17:14:07.956 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: mgrmap e13: x(active, since 61s) 2024-05-03T17:14:07.956 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: overall HEALTH_OK 2024-05-03T17:14:07.956 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:07.956 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:08.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.434+0000 7f0aca39a700 1 -- 172.21.15.82:0/1552292397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aac0035c0 msgr2=0x7f0aac005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:08.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.434+0000 7f0aca39a700 1 --2- 172.21.15.82:0/1552292397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aac0035c0 0x7f0aac005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0ac406a670 tx=0x7f0ab400fe60 comp rx=0 tx=0).stop 2024-05-03T17:14:08.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.434+0000 7f0aca39a700 1 -- 172.21.15.82:0/1552292397 shutdown_connections 2024-05-03T17:14:08.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.434+0000 7f0aca39a700 1 --2- 172.21.15.82:0/1552292397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0aac0035c0 0x7f0aac005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:08.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.434+0000 7f0aca39a700 1 -- 172.21.15.82:0/1552292397 >> 172.21.15.82:0/1552292397 conn(0x7f0ac4078320 msgr2=0x7f0ac4078730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:14:08.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.434+0000 7f0aca39a700 1 -- 172.21.15.82:0/1552292397 shutdown_connections 2024-05-03T17:14:08.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.434+0000 7f0aca39a700 1 -- 172.21.15.82:0/1552292397 wait complete. 2024-05-03T17:14:08.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0aca39a700 1 Processor -- start 2024-05-03T17:14:08.438 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0aca39a700 1 -- start start 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0aca39a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0aac0035c0 0x7f0ac4199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0aca39a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac4199e70 0x7f0ac419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0aca39a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac41a29a0 0x7f0ac41a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0aca39a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac4109dc0 con 0x7f0ac41a29a0 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0aca39a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac4109c40 con 0x7f0ac4199e70 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0aca39a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac4109f40 con 0x7f0aac0035c0 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0ac3fff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0aac0035c0 0x7f0ac4199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0ac3fff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0aac0035c0 0x7f0ac4199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:56362/0 (socket says 172.21.15.82:56362) 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0ac3fff700 1 -- 172.21.15.82:0/2047016301 learned_addr learned my addr 172.21.15.82:0/2047016301 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0ac37fe700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac4199e70 0x7f0ac419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0ac8937700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac41a29a0 0x7f0ac41a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0ac3fff700 1 -- 172.21.15.82:0/2047016301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac4199e70 msgr2=0x7f0ac419a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0ac3fff700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac4199e70 0x7f0ac419a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0ac3fff700 1 -- 172.21.15.82:0/2047016301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac41a29a0 msgr2=0x7f0ac41a4d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:08.439 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0ac3fff700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac41a29a0 0x7f0ac41a4d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:08.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.435+0000 7f0ac3fff700 1 -- 172.21.15.82:0/2047016301 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac41a55a0 con 0x7f0aac0035c0 2024-05-03T17:14:08.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.436+0000 7f0ac3fff700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0aac0035c0 0x7f0ac4199930 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4000c00 tx=0x7f0ab4013b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:14:08.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.436+0000 7f0ac17fa700 1 -- 172.21.15.82:0/2047016301 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab402c070 con 0x7f0aac0035c0 2024-05-03T17:14:08.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.436+0000 7f0ac17fa700 1 -- 172.21.15.82:0/2047016301 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ab4004220 con 0x7f0aac0035c0 2024-05-03T17:14:08.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.436+0000 7f0aca39a700 1 -- 172.21.15.82:0/2047016301 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac41a5830 con 0x7f0aac0035c0 2024-05-03T17:14:08.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.436+0000 7f0ac17fa700 1 -- 172.21.15.82:0/2047016301 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab401cc50 con 0x7f0aac0035c0 2024-05-03T17:14:08.440 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.436+0000 7f0aca39a700 1 -- 172.21.15.82:0/2047016301 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac41a5dc0 con 0x7f0aac0035c0 2024-05-03T17:14:08.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.437+0000 7f0ac17fa700 1 -- 172.21.15.82:0/2047016301 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f0ab402f070 con 0x7f0aac0035c0 2024-05-03T17:14:08.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.437+0000 7f0aca39a700 1 -- 172.21.15.82:0/2047016301 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ac4064b00 con 0x7f0aac0035c0 2024-05-03T17:14:08.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.437+0000 7f0ac17fa700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0ab003e9a0 0x7f0ab0040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:08.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.437+0000 7f0ac17fa700 1 -- 172.21.15.82:0/2047016301 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f0ab4054230 con 0x7f0aac0035c0 2024-05-03T17:14:08.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.437+0000 7f0ac37fe700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0ab003e9a0 0x7f0ab0040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:08.441 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.438+0000 7f0ac37fe700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0ab003e9a0 0x7f0ab0040e60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009ad0 tx=0x7f0ab8009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:14:08.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.442+0000 7f0ac17fa700 1 -- 172.21.15.82:0/2047016301 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f0ab402f360 con 0x7f0aac0035c0 2024-05-03T17:14:08.752 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.751+0000 7f0aca39a700 1 -- 172.21.15.82:0/2047016301 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f0ac41a60c0 con 0x7f0aac0035c0 2024-05-03T17:14:08.753 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:08 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:08.753 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:08 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:08.753 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:08 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:08.753 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:08 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:08.753 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.751+0000 7f0ac17fa700 1 -- 172.21.15.82:0/2047016301 <== mon.1 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1277 (secure 0 0 0) 0x7f0ab4036510 con 0x7f0aac0035c0 2024-05-03T17:14:08.753 INFO:teuthology.orchestra.run.smithi082.stdout: 2024-05-03T17:14:08.753 INFO:teuthology.orchestra.run.smithi082.stdout:{"epoch":3,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","modified":"2024-05-03T17:14:02.406776Z","created":"2024-05-03T17:12:08.010048Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:3300","nonce":0},{"type":"v1","addr":"172.21.15.67:6789","nonce":0}]},"addr":"172.21.15.67:6789/0","public_addr":"172.21.15.67:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:3300","nonce":0},{"type":"v1","addr":"172.21.15.82:6789","nonce":0}]},"addr":"172.21.15.82:6789/0","public_addr":"172.21.15.82:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:3300","nonce":0},{"type":"v1","addr":"172.21.15.70:6789","nonce":0}]},"addr":"172.21.15.70:6789/0","public_addr":"172.21.15.70:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-05-03T17:14:08.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.753+0000 7f0aca39a700 1 -- 172.21.15.82:0/2047016301 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0ab003e9a0 msgr2=0x7f0ab0040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:08.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.753+0000 7f0aca39a700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0ab003e9a0 0x7f0ab0040e60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009ad0 tx=0x7f0ab8009210 comp rx=0 tx=0).stop 2024-05-03T17:14:08.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.753+0000 7f0aca39a700 1 -- 172.21.15.82:0/2047016301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0aac0035c0 msgr2=0x7f0ac4199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:08.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.753+0000 7f0aca39a700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0aac0035c0 0x7f0ac4199930 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4000c00 tx=0x7f0ab4013b90 comp rx=0 tx=0).stop 2024-05-03T17:14:08.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.753+0000 7f0aca39a700 1 -- 172.21.15.82:0/2047016301 shutdown_connections 2024-05-03T17:14:08.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.753+0000 7f0aca39a700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0ab003e9a0 0x7f0ab0040e60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:08.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.753+0000 7f0aca39a700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac41a29a0 0x7f0ac41a4d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:08.755 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.753+0000 7f0aca39a700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0aac0035c0 0x7f0ac4199930 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:08.756 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.753+0000 7f0aca39a700 1 --2- 172.21.15.82:0/2047016301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac4199e70 0x7f0ac419a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:08.756 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.753+0000 7f0aca39a700 1 -- 172.21.15.82:0/2047016301 >> 172.21.15.82:0/2047016301 conn(0x7f0ac4078320 msgr2=0x7f0ac40ffee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:14:08.756 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.753+0000 7f0aca39a700 1 -- 172.21.15.82:0/2047016301 shutdown_connections 2024-05-03T17:14:08.756 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:08.753+0000 7f0aca39a700 1 -- 172.21.15.82:0/2047016301 wait complete. 2024-05-03T17:14:08.756 INFO:teuthology.orchestra.run.smithi082.stderr:dumped monmap epoch 3 2024-05-03T17:14:08.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:08 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:08.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:08 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:08.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:08 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:08.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:08 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:08.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:08 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:08.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:08 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:08.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:08 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:08.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:08 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:14:09.554 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-03T17:14:09.555 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph config generate-minimal-conf 2024-05-03T17:14:09.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:09 smithi082 ceph-mon[33535]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:09.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:09 smithi082 ceph-mon[33535]: from='client.? 172.21.15.82:0/2047016301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T17:14:09.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:09.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:09.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:09.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:09.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:09.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:09.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:09 smithi082 ceph-mon[33535]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-03T17:14:09.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:09 smithi082 ceph-mon[33535]: Updating smithi070:/etc/ceph/ceph.conf 2024-05-03T17:14:09.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:09 smithi082 ceph-mon[33535]: Updating smithi082:/etc/ceph/ceph.conf 2024-05-03T17:14:09.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:09 smithi070 ceph-mon[33250]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:09.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:09 smithi070 ceph-mon[33250]: from='client.? 172.21.15.82:0/2047016301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T17:14:09.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:09.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:09.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:09.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:09.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:09.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:09.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:09 smithi070 ceph-mon[33250]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-03T17:14:09.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:09 smithi070 ceph-mon[33250]: Updating smithi070:/etc/ceph/ceph.conf 2024-05-03T17:14:09.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:09 smithi070 ceph-mon[33250]: Updating smithi082:/etc/ceph/ceph.conf 2024-05-03T17:14:09.809 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:14:09.931 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:09 smithi067 ceph-mon[29731]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:09.931 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:09 smithi067 ceph-mon[29731]: from='client.? 172.21.15.82:0/2047016301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-03T17:14:09.931 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:09.931 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:09.931 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:09.931 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:09.931 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:09.931 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:09.931 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:09 smithi067 ceph-mon[29731]: Updating smithi067:/etc/ceph/ceph.conf 2024-05-03T17:14:09.931 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:09 smithi067 ceph-mon[29731]: Updating smithi070:/etc/ceph/ceph.conf 2024-05-03T17:14:09.931 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:09 smithi067 ceph-mon[29731]: Updating smithi082:/etc/ceph/ceph.conf 2024-05-03T17:14:10.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: Updating smithi067:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: Updating smithi082:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: Updating smithi070:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: Reconfiguring mon.a (unknown last config time)... 2024-05-03T17:14:10.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:14:10.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T17:14:10.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:10.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:10 smithi067 ceph-mon[29731]: Reconfiguring daemon mon.a on smithi067 2024-05-03T17:14:10.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: Updating smithi067:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:14:10.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: Updating smithi082:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:14:10.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: Updating smithi070:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:14:10.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: Reconfiguring mon.a (unknown last config time)... 2024-05-03T17:14:10.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:14:10.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T17:14:10.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:10.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:10 smithi082 ceph-mon[33535]: Reconfiguring daemon mon.a on smithi067 2024-05-03T17:14:10.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: Updating smithi067:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:14:10.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: Updating smithi082:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:14:10.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: Updating smithi070:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/config/ceph.conf 2024-05-03T17:14:10.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:10.811 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: Reconfiguring mon.a (unknown last config time)... 2024-05-03T17:14:10.811 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:14:10.811 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T17:14:10.811 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:10.811 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:10 smithi070 ceph-mon[33250]: Reconfiguring daemon mon.a on smithi067 2024-05-03T17:14:11.939 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:11 smithi067 ceph-mon[29731]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:11.939 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:11 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:11.986 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:11 smithi070 ceph-mon[33250]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:11.986 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:11 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:12.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:11 smithi082 ceph-mon[33535]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:12.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:11 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:12.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.044+0000 7f8b89ece700 1 -- 172.21.15.67:0/3581808254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b84107630 msgr2=0x7f8b84107a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:12.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.044+0000 7f8b89ece700 1 --2- 172.21.15.67:0/3581808254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b84107630 0x7f8b84107a10 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009910 tx=0x7f8b74009c20 comp rx=0 tx=0).stop 2024-05-03T17:14:12.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.044+0000 7f8b89ece700 1 -- 172.21.15.67:0/3581808254 shutdown_connections 2024-05-03T17:14:12.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.044+0000 7f8b89ece700 1 --2- 172.21.15.67:0/3581808254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b84107630 0x7f8b84107a10 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:12.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.044+0000 7f8b89ece700 1 -- 172.21.15.67:0/3581808254 >> 172.21.15.67:0/3581808254 conn(0x7f8b84078520 msgr2=0x7f8b84078930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:14:12.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.044+0000 7f8b89ece700 1 -- 172.21.15.67:0/3581808254 shutdown_connections 2024-05-03T17:14:12.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.044+0000 7f8b89ece700 1 -- 172.21.15.67:0/3581808254 wait complete. 2024-05-03T17:14:12.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.045+0000 7f8b89ece700 1 Processor -- start 2024-05-03T17:14:12.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.045+0000 7f8b89ece700 1 -- start start 2024-05-03T17:14:12.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.045+0000 7f8b89ece700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b84107630 0x7f8b8419d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:12.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.045+0000 7f8b89ece700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8419dc60 0x7f8b841978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.045+0000 7f8b89ece700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b84197e20 0x7f8b841982d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.045+0000 7f8b89ece700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8410baa0 con 0x7f8b8419dc60 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.045+0000 7f8b89ece700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8410b920 con 0x7f8b84107630 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.045+0000 7f8b89ece700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8410bc20 con 0x7f8b84197e20 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.045+0000 7f8b82ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8419dc60 0x7f8b841978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.045+0000 7f8b82ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8419dc60 0x7f8b841978e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36218/0 (socket says 172.21.15.67:36218) 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.045+0000 7f8b82ffd700 1 -- 172.21.15.67:0/2916606721 learned_addr learned my addr 172.21.15.67:0/2916606721 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b83fff700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b84197e20 0x7f8b841982d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b837fe700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b84107630 0x7f8b8419d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b82ffd700 1 -- 172.21.15.67:0/2916606721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b84197e20 msgr2=0x7f8b841982d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b82ffd700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b84197e20 0x7f8b841982d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b82ffd700 1 -- 172.21.15.67:0/2916606721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b84107630 msgr2=0x7f8b8419d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:12.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b82ffd700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b84107630 0x7f8b8419d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b82ffd700 1 -- 172.21.15.67:0/2916606721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b840fe870 con 0x7f8b8419dc60 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b82ffd700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8419dc60 0x7f8b841978e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800b780 tx=0x7f8b7800bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b80ff9700 1 -- 172.21.15.67:0/2916606721 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78004230 con 0x7f8b8419dc60 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b89ece700 1 -- 172.21.15.67:0/2916606721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b840feb60 con 0x7f8b8419dc60 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b80ff9700 1 -- 172.21.15.67:0/2916606721 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b780043d0 con 0x7f8b8419dc60 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b80ff9700 1 -- 172.21.15.67:0/2916606721 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b78011bd0 con 0x7f8b8419dc60 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.046+0000 7f8b89ece700 1 -- 172.21.15.67:0/2916606721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b840ff0a0 con 0x7f8b8419dc60 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.047+0000 7f8b80ff9700 1 -- 172.21.15.67:0/2916606721 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7f8b7801a400 con 0x7f8b8419dc60 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.047+0000 7f8b80ff9700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8b6c03e9a0 0x7f8b6c040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.047+0000 7f8b89ece700 1 -- 172.21.15.67:0/2916606721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b84064b00 con 0x7f8b8419dc60 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.047+0000 7f8b80ff9700 1 -- 172.21.15.67:0/2916606721 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8b780524e0 con 0x7f8b8419dc60 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.047+0000 7f8b837fe700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8b6c03e9a0 0x7f8b6c040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:12.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.048+0000 7f8b837fe700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8b6c03e9a0 0x7f8b6c040e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009750 tx=0x7f8b74005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:14:12.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.053+0000 7f8b80ff9700 1 -- 172.21.15.67:0/2916606721 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f8b78015df0 con 0x7f8b8419dc60 2024-05-03T17:14:12.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.273+0000 7f8b89ece700 1 -- 172.21.15.67:0/2916606721 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f8b840ff360 con 0x7f8b8419dc60 2024-05-03T17:14:12.275 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.274+0000 7f8b80ff9700 1 -- 172.21.15.67:0/2916606721 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+271 (secure 0 0 0) 0x7f8b78003be0 con 0x7f8b8419dc60 2024-05-03T17:14:12.276 INFO:teuthology.orchestra.run.smithi067.stdout:# minimal ceph.conf for 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:14:12.276 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2024-05-03T17:14:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:14:12.276 INFO:teuthology.orchestra.run.smithi067.stdout: mon_host = [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] 2024-05-03T17:14:12.277 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.276+0000 7f8b89ece700 1 -- 172.21.15.67:0/2916606721 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8b6c03e9a0 msgr2=0x7f8b6c040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.276+0000 7f8b89ece700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8b6c03e9a0 0x7f8b6c040e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009750 tx=0x7f8b74005fd0 comp rx=0 tx=0).stop 2024-05-03T17:14:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.276+0000 7f8b89ece700 1 -- 172.21.15.67:0/2916606721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8419dc60 msgr2=0x7f8b841978e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.276+0000 7f8b89ece700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8419dc60 0x7f8b841978e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8b7800b780 tx=0x7f8b7800bb40 comp rx=0 tx=0).stop 2024-05-03T17:14:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.276+0000 7f8b89ece700 1 -- 172.21.15.67:0/2916606721 shutdown_connections 2024-05-03T17:14:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.276+0000 7f8b89ece700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8b6c03e9a0 0x7f8b6c040e60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.276+0000 7f8b89ece700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8b84197e20 0x7f8b841982d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.276+0000 7f8b89ece700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8b84107630 0x7f8b8419d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.276+0000 7f8b89ece700 1 --2- 172.21.15.67:0/2916606721 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8b8419dc60 0x7f8b841978e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.276+0000 7f8b89ece700 1 -- 172.21.15.67:0/2916606721 >> 172.21.15.67:0/2916606721 conn(0x7f8b84078520 msgr2=0x7f8b84100210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:14:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.276+0000 7f8b89ece700 1 -- 172.21.15.67:0/2916606721 shutdown_connections 2024-05-03T17:14:12.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:12.276+0000 7f8b89ece700 1 -- 172.21.15.67:0/2916606721 wait complete. 2024-05-03T17:14:12.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:12 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2916606721' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:13.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:12 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2916606721' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:13.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:12 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2916606721' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:13.173 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-03T17:14:13.173 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:14:13.174 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.conf 2024-05-03T17:14:13.200 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:14:13.200 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T17:14:13.265 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-05-03T17:14:13.265 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.conf 2024-05-03T17:14:13.292 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-05-03T17:14:13.292 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T17:14:13.355 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T17:14:13.355 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.conf 2024-05-03T17:14:13.386 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T17:14:13.386 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-03T17:14:13.454 INFO:tasks.cephadm:Adding mgr.x on smithi067 2024-05-03T17:14:13.454 INFO:tasks.cephadm:Adding mgr.y on smithi070 2024-05-03T17:14:13.454 INFO:tasks.cephadm:Adding mgr.z on smithi082 2024-05-03T17:14:13.454 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch apply mgr '3;smithi067=x;smithi070=y;smithi082=z' 2024-05-03T17:14:13.728 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:14:13.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:13 smithi067 ceph-mon[29731]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:13.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:13.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:13.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:13 smithi067 ceph-mon[29731]: Reconfiguring mon.b (monmap changed)... 2024-05-03T17:14:13.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:14:13.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T17:14:13.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:13.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:13 smithi067 ceph-mon[29731]: Reconfiguring daemon mon.b on smithi070 2024-05-03T17:14:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:13 smithi070 ceph-mon[33250]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:13 smithi070 ceph-mon[33250]: Reconfiguring mon.b (monmap changed)... 2024-05-03T17:14:14.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:14:14.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T17:14:14.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:14.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:13 smithi070 ceph-mon[33250]: Reconfiguring daemon mon.b on smithi070 2024-05-03T17:14:14.153 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:13 smithi082 ceph-mon[33535]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:14.153 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:14.153 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:14.153 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:13 smithi082 ceph-mon[33535]: Reconfiguring mon.b (monmap changed)... 2024-05-03T17:14:14.154 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:14:14.154 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T17:14:14.154 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:14.154 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:13 smithi082 ceph-mon[33535]: Reconfiguring daemon mon.b on smithi070 2024-05-03T17:14:15.172 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.167+0000 7fab7c95e700 1 -- 172.21.15.82:0/353278642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab78103b70 msgr2=0x7fab78105f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:15.172 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.167+0000 7fab7c95e700 1 --2- 172.21.15.82:0/353278642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab78103b70 0x7fab78105f60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fab68009910 tx=0x7fab68009c20 comp rx=0 tx=0).stop 2024-05-03T17:14:15.172 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.167+0000 7fab7c95e700 1 -- 172.21.15.82:0/353278642 shutdown_connections 2024-05-03T17:14:15.172 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.167+0000 7fab7c95e700 1 --2- 172.21.15.82:0/353278642 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab78103b70 0x7fab78105f60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:15.172 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.167+0000 7fab7c95e700 1 -- 172.21.15.82:0/353278642 >> 172.21.15.82:0/353278642 conn(0x7fab780fd4e0 msgr2=0x7fab780ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:14:15.172 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.167+0000 7fab7c95e700 1 -- 172.21.15.82:0/353278642 shutdown_connections 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.167+0000 7fab7c95e700 1 -- 172.21.15.82:0/353278642 wait complete. 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.168+0000 7fab7c95e700 1 Processor -- start 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.168+0000 7fab7c95e700 1 -- start start 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.168+0000 7fab7c95e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab78103b70 0x7fab781034d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.168+0000 7fab7c95e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab78101b20 0x7fab78101fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.168+0000 7fab7c95e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab781024e0 0x7fab781a27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.168+0000 7fab7c95e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fab78108f80 con 0x7fab781024e0 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.168+0000 7fab7c95e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fab78108e00 con 0x7fab78103b70 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.168+0000 7fab7c95e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fab78109100 con 0x7fab78101b20 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab76d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab781024e0 0x7fab781a27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab76d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab781024e0 0x7fab781a27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.82:60194/0 (socket says 172.21.15.82:60194) 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab76d9d700 1 -- 172.21.15.82:0/3439889037 learned_addr learned my addr 172.21.15.82:0/3439889037 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab6ffff700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab78101b20 0x7fab78101fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab7659c700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab78103b70 0x7fab781034d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab76d9d700 1 -- 172.21.15.82:0/3439889037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab78101b20 msgr2=0x7fab78101fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:15.173 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab76d9d700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab78101b20 0x7fab78101fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab76d9d700 1 -- 172.21.15.82:0/3439889037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab78103b70 msgr2=0x7fab781034d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab76d9d700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab78103b70 0x7fab781034d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab76d9d700 1 -- 172.21.15.82:0/3439889037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab781a2d90 con 0x7fab781024e0 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab7659c700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab78103b70 0x7fab781034d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab76d9d700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab781024e0 0x7fab781a27d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fab7000ba50 tx=0x7fab7000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab6f7fe700 1 -- 172.21.15.82:0/3439889037 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab70019070 con 0x7fab781024e0 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab6f7fe700 1 -- 172.21.15.82:0/3439889037 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab70014440 con 0x7fab781024e0 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab7c95e700 1 -- 172.21.15.82:0/3439889037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab781a3080 con 0x7fab781024e0 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab6f7fe700 1 -- 172.21.15.82:0/3439889037 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab7000ce60 con 0x7fab781024e0 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.169+0000 7fab7c95e700 1 -- 172.21.15.82:0/3439889037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab781a3640 con 0x7fab781024e0 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.170+0000 7fab7c95e700 1 -- 172.21.15.82:0/3439889037 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab78064b00 con 0x7fab781024e0 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.170+0000 7fab6f7fe700 1 -- 172.21.15.82:0/3439889037 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fab7001c070 con 0x7fab781024e0 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.171+0000 7fab6f7fe700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fab6403e9a0 0x7fab64040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:15.174 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.171+0000 7fab6f7fe700 1 -- 172.21.15.82:0/3439889037 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fab70051d90 con 0x7fab781024e0 2024-05-03T17:14:15.175 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.171+0000 7fab7659c700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fab6403e9a0 0x7fab64040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:15.175 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.171+0000 7fab7659c700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fab6403e9a0 0x7fab64040e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fab68009750 tx=0x7fab6801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:14:15.177 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.176+0000 7fab6f7fe700 1 -- 172.21.15.82:0/3439889037 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fab7001c360 con 0x7fab781024e0 2024-05-03T17:14:15.404 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.403+0000 7fab7c95e700 1 -- 172.21.15.82:0/3439889037 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi067=x;smithi070=y;smithi082=z", "target": ["mon-mgr", ""]}) -- 0x7fab780638e0 con 0x7fab6403e9a0 2024-05-03T17:14:15.412 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.410+0000 7fab6f7fe700 1 -- 172.21.15.82:0/3439889037 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fab780638e0 con 0x7fab6403e9a0 2024-05-03T17:14:15.412 INFO:teuthology.orchestra.run.smithi082.stdout:Scheduled mgr update... 2024-05-03T17:14:15.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.413+0000 7fab7c95e700 1 -- 172.21.15.82:0/3439889037 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fab6403e9a0 msgr2=0x7fab64040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:15.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.413+0000 7fab7c95e700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fab6403e9a0 0x7fab64040e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fab68009750 tx=0x7fab6801a040 comp rx=0 tx=0).stop 2024-05-03T17:14:15.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.413+0000 7fab7c95e700 1 -- 172.21.15.82:0/3439889037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab781024e0 msgr2=0x7fab781a27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:15.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.413+0000 7fab7c95e700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab781024e0 0x7fab781a27d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fab7000ba50 tx=0x7fab7000bd60 comp rx=0 tx=0).stop 2024-05-03T17:14:15.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.413+0000 7fab7c95e700 1 -- 172.21.15.82:0/3439889037 shutdown_connections 2024-05-03T17:14:15.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.413+0000 7fab7c95e700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fab6403e9a0 0x7fab64040e60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:15.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.413+0000 7fab7c95e700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fab781024e0 0x7fab781a27d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:15.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.413+0000 7fab7c95e700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fab78103b70 0x7fab781034d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:15.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.413+0000 7fab7c95e700 1 --2- 172.21.15.82:0/3439889037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fab78101b20 0x7fab78101fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:15.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.413+0000 7fab7c95e700 1 -- 172.21.15.82:0/3439889037 >> 172.21.15.82:0/3439889037 conn(0x7fab780fd4e0 msgr2=0x7fab780fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:14:15.415 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.413+0000 7fab7c95e700 1 -- 172.21.15.82:0/3439889037 shutdown_connections 2024-05-03T17:14:15.416 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:14:15.413+0000 7fab7c95e700 1 -- 172.21.15.82:0/3439889037 wait complete. 2024-05-03T17:14:15.554 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:15 smithi082 ceph-mon[33535]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:15.555 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:15 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:15.555 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:15 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:15.555 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:15 smithi082 ceph-mon[33535]: Reconfiguring mon.c (monmap changed)... 2024-05-03T17:14:15.555 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:15 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:14:15.555 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:15 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T17:14:15.555 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:15 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:15.555 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:15 smithi082 ceph-mon[33535]: Reconfiguring daemon mon.c on smithi082 2024-05-03T17:14:15.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:15 smithi067 ceph-mon[29731]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:15.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:15 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:15.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:15 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:15.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:15 smithi067 ceph-mon[29731]: Reconfiguring mon.c (monmap changed)... 2024-05-03T17:14:15.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:15 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:14:15.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:15 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T17:14:15.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:15 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:15.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:15 smithi067 ceph-mon[29731]: Reconfiguring daemon mon.c on smithi082 2024-05-03T17:14:15.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:15 smithi070 ceph-mon[33250]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:15.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:15 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:15.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:15 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:15.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:15 smithi070 ceph-mon[33250]: Reconfiguring mon.c (monmap changed)... 2024-05-03T17:14:15.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:15 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-03T17:14:15.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:15 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-03T17:14:15.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:15 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:15.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:15 smithi070 ceph-mon[33250]: Reconfiguring daemon mon.c on smithi082 2024-05-03T17:14:16.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:16 smithi067 ceph-mon[29731]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi067=x;smithi070=y;smithi082=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:14:16.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:16 smithi067 ceph-mon[29731]: Saving service mgr spec with placement smithi067=x;smithi070=y;smithi082=z;count:3 2024-05-03T17:14:16.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:16 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:16.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:16 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:16.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:16 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:16.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:16 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:16.727 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:16 smithi082 ceph-mon[33535]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi067=x;smithi070=y;smithi082=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:14:16.728 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:16 smithi082 ceph-mon[33535]: Saving service mgr spec with placement smithi067=x;smithi070=y;smithi082=z;count:3 2024-05-03T17:14:16.728 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:16 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:16.728 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:16 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:16.728 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:16 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:16.728 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:16 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:16.761 DEBUG:teuthology.orchestra.run.smithi070:mgr.y> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.y.service 2024-05-03T17:14:16.764 DEBUG:teuthology.orchestra.run.smithi082:mgr.z> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.z.service 2024-05-03T17:14:16.766 INFO:tasks.cephadm:Deploying OSDs... 2024-05-03T17:14:16.767 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:14:16.767 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2024-05-03T17:14:16.783 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-03T17:14:16.783 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2024-05-03T17:14:16.787 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:16 smithi070 ceph-mon[33250]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi067=x;smithi070=y;smithi082=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:14:16.787 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:16 smithi070 ceph-mon[33250]: Saving service mgr spec with placement smithi067=x;smithi070=y;smithi082=z;count:3 2024-05-03T17:14:16.787 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:16 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:16.787 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:16 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:16.787 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:16 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:16.787 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:16 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:16.839 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-03T17:14:16.840 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T17:14:16.840 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-05-03T17:14:16.840 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T17:14:16.840 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T17:14:16.840 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-03 17:13:19.341998411 +0000 2024-05-03T17:14:16.840 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-03 17:08:30.725741743 +0000 2024-05-03T17:14:16.840 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-03 17:08:30.725741743 +0000 2024-05-03T17:14:16.840 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-03 17:08:30.725741743 +0000 2024-05-03T17:14:16.840 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-03T17:14:16.903 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-03T17:14:16.904 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-03T17:14:16.904 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000198514 s, 2.6 MB/s 2024-05-03T17:14:16.905 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-03T17:14:16.962 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2024-05-03T17:14:17.018 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-03T17:14:17.018 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T17:14:17.018 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-03T17:14:17.018 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T17:14:17.018 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T17:14:17.019 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-03 17:13:19.378997651 +0000 2024-05-03T17:14:17.019 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-03 17:08:31.073734380 +0000 2024-05-03T17:14:17.019 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-03 17:08:31.073734380 +0000 2024-05-03T17:14:17.019 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-03 17:08:31.073734380 +0000 2024-05-03T17:14:17.019 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-03T17:14:17.081 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-03T17:14:17.082 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-03T17:14:17.082 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000189686 s, 2.7 MB/s 2024-05-03T17:14:17.083 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-03T17:14:17.139 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2024-05-03T17:14:17.194 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-03T17:14:17.194 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T17:14:17.194 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-03T17:14:17.194 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T17:14:17.194 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T17:14:17.194 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-03 17:13:19.413996933 +0000 2024-05-03T17:14:17.194 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-03 17:08:31.409727270 +0000 2024-05-03T17:14:17.194 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-03 17:08:31.409727270 +0000 2024-05-03T17:14:17.195 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-03 17:08:31.409727270 +0000 2024-05-03T17:14:17.195 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-03T17:14:17.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-03T17:14:17.257 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-03T17:14:17.258 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000188219 s, 2.7 MB/s 2024-05-03T17:14:17.259 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-03T17:14:17.316 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2024-05-03T17:14:17.371 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-03T17:14:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T17:14:17.372 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-03T17:14:17.372 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T17:14:17.372 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T17:14:17.372 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2024-05-03 17:13:19.448996214 +0000 2024-05-03T17:14:17.372 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2024-05-03 17:08:31.752720011 +0000 2024-05-03T17:14:17.372 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2024-05-03 17:08:31.752720011 +0000 2024-05-03T17:14:17.372 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: 2024-05-03 17:08:31.752720011 +0000 2024-05-03T17:14:17.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-03T17:14:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2024-05-03T17:14:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2024-05-03T17:14:17.436 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000175443 s, 2.9 MB/s 2024-05-03T17:14:17.438 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-03T17:14:17.493 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-05-03T17:14:17.494 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-05-03T17:14:17.512 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-03T17:14:17.512 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-05-03T17:14:17.570 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-03T17:14:17.570 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T17:14:17.570 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-03T17:14:17.570 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T17:14:17.570 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T17:14:17.570 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-05-03 17:14:01.834305238 +0000 2024-05-03T17:14:17.570 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-05-03 17:08:41.909329822 +0000 2024-05-03T17:14:17.570 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-05-03 17:08:41.909329822 +0000 2024-05-03T17:14:17.570 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-05-03 17:08:41.909329822 +0000 2024-05-03T17:14:17.571 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-03T17:14:17.636 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-05-03T17:14:17.637 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-05-03T17:14:17.637 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000132186 s, 3.9 MB/s 2024-05-03T17:14:17.638 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-03T17:14:17.650 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:17 smithi082 ceph-mon[33535]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:17.651 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:17.651 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:17.651 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:17.651 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T17:14:17.651 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T17:14:17.651 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T17:14:17.651 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:17.651 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:17 smithi082 ceph-mon[33535]: Deploying daemon mgr.z on smithi082 2024-05-03T17:14:17.696 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-05-03T17:14:17.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:17 smithi067 ceph-mon[29731]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:17.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:17.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:17.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:17.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T17:14:17.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T17:14:17.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T17:14:17.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:17.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:17 smithi067 ceph-mon[29731]: Deploying daemon mgr.z on smithi082 2024-05-03T17:14:17.753 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-03T17:14:17.753 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T17:14:17.753 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-05-03T17:14:17.753 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T17:14:17.753 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T17:14:17.753 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-05-03 17:14:01.835305210 +0000 2024-05-03T17:14:17.753 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-05-03 17:08:42.240320382 +0000 2024-05-03T17:14:17.753 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-05-03 17:08:42.240320382 +0000 2024-05-03T17:14:17.753 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-05-03 17:08:42.240320382 +0000 2024-05-03T17:14:17.753 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-03T17:14:17.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:17 smithi070 ceph-mon[33250]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:17.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:17.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:17.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:17.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T17:14:17.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T17:14:17.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T17:14:17.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:17.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:17 smithi070 ceph-mon[33250]: Deploying daemon mgr.z on smithi082 2024-05-03T17:14:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-05-03T17:14:17.817 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-05-03T17:14:17.818 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.00011995 s, 4.3 MB/s 2024-05-03T17:14:17.819 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-03T17:14:17.876 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-05-03T17:14:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-03T17:14:17.932 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T17:14:17.933 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-03T17:14:17.933 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T17:14:17.933 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T17:14:17.933 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-05-03 17:14:01.835305210 +0000 2024-05-03T17:14:17.933 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-05-03 17:08:42.561311228 +0000 2024-05-03T17:14:17.933 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-05-03 17:08:42.561311228 +0000 2024-05-03T17:14:17.933 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-05-03 17:08:42.561311228 +0000 2024-05-03T17:14:17.933 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-03T17:14:17.995 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-05-03T17:14:17.995 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-05-03T17:14:17.995 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000117738 s, 4.3 MB/s 2024-05-03T17:14:17.997 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-03T17:14:18.054 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-05-03T17:14:18.110 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-03T17:14:18.110 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T17:14:18.110 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-03T17:14:18.110 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T17:14:18.110 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T17:14:18.110 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-05-03 17:14:01.836305181 +0000 2024-05-03T17:14:18.111 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-05-03 17:08:42.885301988 +0000 2024-05-03T17:14:18.111 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-05-03 17:08:42.885301988 +0000 2024-05-03T17:14:18.111 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-05-03 17:08:42.885301988 +0000 2024-05-03T17:14:18.111 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-03T17:14:18.174 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-05-03T17:14:18.174 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-05-03T17:14:18.174 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 9.6407e-05 s, 5.3 MB/s 2024-05-03T17:14:18.175 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-03T17:14:18.231 DEBUG:teuthology.orchestra.run.smithi082:> set -ex 2024-05-03T17:14:18.231 DEBUG:teuthology.orchestra.run.smithi082:> dd if=/scratch_devs of=/dev/stdout 2024-05-03T17:14:18.247 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-03T17:14:18.247 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_1 2024-05-03T17:14:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-03T17:14:18.302 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T17:14:18.302 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-03T17:14:18.302 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T17:14:18.302 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T17:14:18.302 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-05-03 17:13:52.335280352 +0000 2024-05-03T17:14:18.302 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-05-03 17:08:46.964218275 +0000 2024-05-03T17:14:18.303 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-05-03 17:08:46.964218275 +0000 2024-05-03T17:14:18.303 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-05-03 17:08:46.964218275 +0000 2024-05-03T17:14:18.303 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-03T17:14:18.366 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-05-03T17:14:18.366 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-05-03T17:14:18.366 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000271592 s, 1.9 MB/s 2024-05-03T17:14:18.367 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-03T17:14:18.423 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_2 2024-05-03T17:14:18.478 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-03T17:14:18.478 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T17:14:18.478 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-03T17:14:18.478 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T17:14:18.478 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T17:14:18.478 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-05-03 17:13:52.335280352 +0000 2024-05-03T17:14:18.478 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-05-03 17:08:47.289211935 +0000 2024-05-03T17:14:18.478 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-05-03 17:08:47.289211935 +0000 2024-05-03T17:14:18.478 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-05-03 17:08:47.289211935 +0000 2024-05-03T17:14:18.479 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-03T17:14:18.541 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-05-03T17:14:18.541 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-05-03T17:14:18.541 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000169305 s, 3.0 MB/s 2024-05-03T17:14:18.542 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-03T17:14:18.601 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_3 2024-05-03T17:14:18.657 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-03T17:14:18.657 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T17:14:18.658 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-03T17:14:18.658 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T17:14:18.658 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T17:14:18.658 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-05-03 17:13:52.336280333 +0000 2024-05-03T17:14:18.658 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-05-03 17:08:47.624205400 +0000 2024-05-03T17:14:18.658 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-05-03 17:08:47.624205400 +0000 2024-05-03T17:14:18.658 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-05-03 17:08:47.624205400 +0000 2024-05-03T17:14:18.658 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-03T17:14:18.733 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-05-03T17:14:18.733 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-05-03T17:14:18.733 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000186257 s, 2.7 MB/s 2024-05-03T17:14:18.735 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-03T17:14:18.791 DEBUG:teuthology.orchestra.run.smithi082:> stat /dev/vg_nvme/lv_4 2024-05-03T17:14:18.847 INFO:teuthology.orchestra.run.smithi082.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-03T17:14:18.847 INFO:teuthology.orchestra.run.smithi082.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-03T17:14:18.847 INFO:teuthology.orchestra.run.smithi082.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-03T17:14:18.847 INFO:teuthology.orchestra.run.smithi082.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-03T17:14:18.847 INFO:teuthology.orchestra.run.smithi082.stdout:Context: system_u:object_r:device_t:s0 2024-05-03T17:14:18.847 INFO:teuthology.orchestra.run.smithi082.stdout:Access: 2024-05-03 17:14:18.684781531 +0000 2024-05-03T17:14:18.848 INFO:teuthology.orchestra.run.smithi082.stdout:Modify: 2024-05-03 17:08:47.971198632 +0000 2024-05-03T17:14:18.848 INFO:teuthology.orchestra.run.smithi082.stdout:Change: 2024-05-03 17:08:47.971198632 +0000 2024-05-03T17:14:18.848 INFO:teuthology.orchestra.run.smithi082.stdout: Birth: 2024-05-03 17:08:47.971198632 +0000 2024-05-03T17:14:18.848 DEBUG:teuthology.orchestra.run.smithi082:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-03T17:14:18.911 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records in 2024-05-03T17:14:18.912 INFO:teuthology.orchestra.run.smithi082.stderr:1+0 records out 2024-05-03T17:14:18.912 INFO:teuthology.orchestra.run.smithi082.stderr:512 bytes copied, 0.000192182 s, 2.7 MB/s 2024-05-03T17:14:18.913 DEBUG:teuthology.orchestra.run.smithi082:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-03T17:14:18.971 INFO:tasks.cephadm:Deploying osd.0 on smithi067 with /dev/vg_nvme/lv_4... 2024-05-03T17:14:18.971 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-03T17:14:19.014 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:18 smithi082 systemd[1]: Starting Ceph mgr.z for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:14:19.221 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:14:19.419 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:19 smithi082 podman[35544]: 2024-05-03 17:14:19.045553001 +0000 UTC m=+0.017205349 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:14:19.419 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:19 smithi082 podman[35544]: 2024-05-03 17:14:19.161363552 +0000 UTC m=+0.133015876 container create d4f248973a3ef498632a528ef85f6d7fc0b27cbbcfa65efa34776d23824ad469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-03T17:14:19.600 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:19 smithi067 ceph-mon[29731]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:19.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:19 smithi082 ceph-mon[33535]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:19.765 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:19 smithi082 podman[35544]: 2024-05-03 17:14:19.464177773 +0000 UTC m=+0.435830107 container init d4f248973a3ef498632a528ef85f6d7fc0b27cbbcfa65efa34776d23824ad469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-03T17:14:19.765 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:19 smithi082 podman[35544]: 2024-05-03 17:14:19.467105735 +0000 UTC m=+0.438758060 container start d4f248973a3ef498632a528ef85f6d7fc0b27cbbcfa65efa34776d23824ad469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-05-03T17:14:19.765 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:19 smithi082 ceph-mgr[35563]: -- 172.21.15.82:0/4247286419 <== mon.1 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5612393d05a0 con 0x5612393a5000 2024-05-03T17:14:19.765 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:19 smithi082 bash[35544]: d4f248973a3ef498632a528ef85f6d7fc0b27cbbcfa65efa34776d23824ad469 2024-05-03T17:14:19.765 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:19 smithi082 systemd[1]: Started Ceph mgr.z for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:14:19.765 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:19 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:19.630+0000 7f457a639200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T17:14:19.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:19 smithi070 ceph-mon[33250]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:20.220 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:19 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:19.812+0000 7f457a639200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T17:14:20.220 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:19 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:19.912+0000 7f457a639200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T17:14:20.221 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:20 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:20.219+0000 7f457a639200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T17:14:20.514 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:20 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:20.303+0000 7f457a639200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T17:14:20.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:20 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:20.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:20 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:20.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:20 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:20.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:20 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T17:14:20.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:20 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T17:14:20.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:20 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T17:14:20.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:20 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:20.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:20 smithi070 ceph-mon[33250]: Deploying daemon mgr.y on smithi070 2024-05-03T17:14:20.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:20 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:20.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:20 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:20.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:20 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:20.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:20 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T17:14:20.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:20 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T17:14:20.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:20 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T17:14:20.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:20 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:20.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:20 smithi067 ceph-mon[29731]: Deploying daemon mgr.y on smithi070 2024-05-03T17:14:21.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:20 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:21.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:20 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:21.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:20 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:21.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:20 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T17:14:21.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:20 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-03T17:14:21.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:20 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T17:14:21.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:20 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:21.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:20 smithi082 ceph-mon[33535]: Deploying daemon mgr.y on smithi070 2024-05-03T17:14:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-03T17:14:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-03T17:14:21.171 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-05-03T17:14:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-05-03T17:14:21.171 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122598 s, 855 MB/s 2024-05-03T17:14:21.172 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-05-03T17:14:21.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:21 smithi082 ceph-mon[33535]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:21.765 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:21 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:21.491+0000 7f457a639200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T17:14:21.765 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:21 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:21.660+0000 7f457a639200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T17:14:21.793 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:21 smithi067 ceph-mon[29731]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:21.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_4 2024-05-03T17:14:21.910 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:21 smithi070 ceph-mon[33250]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:22.072 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:14:22.117 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:21 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:21.829+0000 7f457a639200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T17:14:22.117 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:22 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:22.116+0000 7f457a639200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T17:14:22.308 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:22.169+0000 7f366f0c3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-03T17:14:22.418 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:22 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:22.209+0000 7f457a639200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T17:14:22.418 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:22 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:22.417+0000 7f457a639200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T17:14:22.764 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:22 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:22.617+0000 7f457a639200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T17:14:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:22.358+0000 7f366f0c3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-03T17:14:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:22.466+0000 7f366f0c3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-03T17:14:22.793 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:22.792+0000 7f366f0c3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-03T17:14:23.058 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:22.885+0000 7f366f0c3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-03T17:14:23.434 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:23.435 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:23.435 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:23.435 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:23.435 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:23 smithi070 ceph-mon[33250]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:23.435 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:23.443 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:23.443 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:23.444 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:23.444 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:23.444 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:23 smithi067 ceph-mon[29731]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:23.444 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:23.477 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:23 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:23.248+0000 7f457a639200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T17:14:23.477 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:23 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:23.476+0000 7f457a639200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-03T17:14:23.477 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:23.477 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:23.477 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:23.477 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:23.478 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:23 smithi082 ceph-mon[33535]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:23.478 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.680+0000 7fada1597700 1 -- 172.21.15.67:0/197039044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9c103ff0 msgr2=0x7fad9c1063e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.680+0000 7fada1597700 1 --2- 172.21.15.67:0/197039044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9c103ff0 0x7fad9c1063e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fad8c009990 tx=0x7fad8c009ca0 comp rx=0 tx=0).stop 2024-05-03T17:14:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.680+0000 7fada1597700 1 -- 172.21.15.67:0/197039044 shutdown_connections 2024-05-03T17:14:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.680+0000 7fada1597700 1 --2- 172.21.15.67:0/197039044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9c109250 0x7fad9c10b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.680+0000 7fada1597700 1 --2- 172.21.15.67:0/197039044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9c103ff0 0x7fad9c1063e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.680+0000 7fada1597700 1 --2- 172.21.15.67:0/197039044 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad9c106920 0x7fad9c108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.680+0000 7fada1597700 1 -- 172.21.15.67:0/197039044 >> 172.21.15.67:0/197039044 conn(0x7fad9c0fd9a0 msgr2=0x7fad9c0ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:14:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.680+0000 7fada1597700 1 -- 172.21.15.67:0/197039044 shutdown_connections 2024-05-03T17:14:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.680+0000 7fada1597700 1 -- 172.21.15.67:0/197039044 wait complete. 2024-05-03T17:14:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.681+0000 7fada1597700 1 Processor -- start 2024-05-03T17:14:23.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.681+0000 7fada1597700 1 -- start start 2024-05-03T17:14:23.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.681+0000 7fada1597700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9c103ff0 0x7fad9c19b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:23.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.681+0000 7fada1597700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9c106920 0x7fad9c19be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.681+0000 7fada1597700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad9c109250 0x7fad9c1a2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.681+0000 7fada1597700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c10efc0 con 0x7fad9c106920 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.681+0000 7fada1597700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c10ee40 con 0x7fad9c103ff0 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.681+0000 7fada1597700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fad9c10f140 con 0x7fad9c109250 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9c106920 0x7fad9c19be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9c103ff0 0x7fad9c19b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9b7fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad9c109250 0x7fad9c1a2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9b7fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad9c109250 0x7fad9c1a2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:46398/0 (socket says 172.21.15.67:46398) 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9a7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9c106920 0x7fad9c19be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54700/0 (socket says 172.21.15.67:54700) 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9b7fe700 1 -- 172.21.15.67:0/4160988873 learned_addr learned my addr 172.21.15.67:0/4160988873 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9affd700 1 -- 172.21.15.67:0/4160988873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad9c109250 msgr2=0x7fad9c1a2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9affd700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad9c109250 0x7fad9c1a2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9affd700 1 -- 172.21.15.67:0/4160988873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9c106920 msgr2=0x7fad9c19be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:23.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9affd700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9c106920 0x7fad9c19be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9affd700 1 -- 172.21.15.67:0/4160988873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad9c1a3580 con 0x7fad9c103ff0 2024-05-03T17:14:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9b7fe700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad9c109250 0x7fad9c1a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:14:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9a7fc700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9c106920 0x7fad9c19be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:14:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.682+0000 7fad9affd700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9c103ff0 0x7fad9c19b920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00b650 tx=0x7fad8c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:14:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.683+0000 7fad83fff700 1 -- 172.21.15.67:0/4160988873 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c026070 con 0x7fad9c103ff0 2024-05-03T17:14:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.683+0000 7fad83fff700 1 -- 172.21.15.67:0/4160988873 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad8c021410 con 0x7fad9c103ff0 2024-05-03T17:14:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.683+0000 7fad83fff700 1 -- 172.21.15.67:0/4160988873 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad8c017650 con 0x7fad9c103ff0 2024-05-03T17:14:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.683+0000 7fada1597700 1 -- 172.21.15.67:0/4160988873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad9c1a3810 con 0x7fad9c103ff0 2024-05-03T17:14:23.686 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.683+0000 7fada1597700 1 -- 172.21.15.67:0/4160988873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad9c1a3c60 con 0x7fad9c103ff0 2024-05-03T17:14:23.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.684+0000 7fad83fff700 1 -- 172.21.15.67:0/4160988873 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 13) ==== 50278+0+0 (secure 0 0 0) 0x7fad8c029070 con 0x7fad9c103ff0 2024-05-03T17:14:23.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.684+0000 7fada1597700 1 -- 172.21.15.67:0/4160988873 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad9c06a190 con 0x7fad9c103ff0 2024-05-03T17:14:23.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.685+0000 7fad83fff700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fad8403e750 0x7fad84040c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:23.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.685+0000 7fad83fff700 1 -- 172.21.15.67:0/4160988873 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fad8c053790 con 0x7fad9c103ff0 2024-05-03T17:14:23.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.685+0000 7fad9a7fc700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fad8403e750 0x7fad84040c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:23.687 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.685+0000 7fad9a7fc700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fad8403e750 0x7fad84040c10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fad90011490 tx=0x7fad90009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:14:23.690 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.690+0000 7fad83fff700 1 -- 172.21.15.67:0/4160988873 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fad8c013070 con 0x7fad9c103ff0 2024-05-03T17:14:23.923 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:23.923+0000 7fada1597700 1 -- 172.21.15.67:0/4160988873 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fad9c0638e0 con 0x7fad8403e750 2024-05-03T17:14:24.264 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:23 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:23.916+0000 7f457a639200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T17:14:24.265 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:24 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:24.011+0000 7f457a639200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T17:14:24.465 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:24 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:24.124+0000 7f366f0c3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-03T17:14:24.465 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:24 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:24.293+0000 7f366f0c3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-03T17:14:24.465 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:24 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:24.464+0000 7f366f0c3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-03T17:14:24.678 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:24 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:24.397+0000 7f457a639200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-03T17:14:24.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:24.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:24.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:24.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:24.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:24.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:24.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:24.742 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: Reconfiguring mgr.x (unknown last config time)... 2024-05-03T17:14:24.743 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T17:14:24.743 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T17:14:24.743 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:24.743 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: Reconfiguring daemon mgr.x on smithi067 2024-05-03T17:14:24.743 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:14:24.743 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:14:24.743 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:14:24.743 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:24.753 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:24.753 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:24.753 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:24.753 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:24.753 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:24.753 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:24.753 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:24.753 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: Reconfiguring mgr.x (unknown last config time)... 2024-05-03T17:14:24.753 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T17:14:24.753 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T17:14:24.753 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:24.753 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: Reconfiguring daemon mgr.x on smithi067 2024-05-03T17:14:24.754 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:14:24.754 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:14:24.754 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:14:24.754 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:25.014 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:24 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:24.749+0000 7f457a639200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T17:14:25.015 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:24 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:24.839+0000 7f457a639200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T17:14:25.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:25.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:25.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:25.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:25.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:25.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:25.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:25.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: Reconfiguring mgr.x (unknown last config time)... 2024-05-03T17:14:25.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-03T17:14:25.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-03T17:14:25.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:25.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: Reconfiguring daemon mgr.x on smithi067 2024-05-03T17:14:25.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='client.24116 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:14:25.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:14:25.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:14:25.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:25.046 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:24 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:24.752+0000 7f366f0c3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-03T17:14:25.046 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:24 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:24.845+0000 7f366f0c3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-03T17:14:25.046 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:25 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:25.045+0000 7f366f0c3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-03T17:14:25.308 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:25 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:25.235+0000 7f366f0c3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-03T17:14:25.514 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:25.079+0000 7f457a639200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T17:14:25.514 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:25.211+0000 7f457a639200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T17:14:26.015 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:25.517+0000 7f457a639200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T17:14:26.015 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:25.632+0000 7f457a639200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T17:14:26.015 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:25.723+0000 7f457a639200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T17:14:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:25 smithi082 ceph-mon[33535]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:26.053 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:25 smithi070 ceph-mon[33250]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:26.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:26.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:26.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:26.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:26.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:26.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:26.054 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:25 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:25.831+0000 7f366f0c3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-03T17:14:26.054 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:26.052+0000 7f366f0c3200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-03T17:14:26.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:25 smithi067 ceph-mon[29731]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:26.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:26.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:26.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:26.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:26.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:26.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:26.767 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:26 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:26.311+0000 7f457a639200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T17:14:26.767 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:14:26 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z[35558]: 2024-05-03T17:14:26.525+0000 7f457a639200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T17:14:26.808 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:26.475+0000 7f366f0c3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-03T17:14:26.808 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:26.571+0000 7f366f0c3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-03T17:14:27.301 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:26.948+0000 7f366f0c3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-03T17:14:27.301 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:27.300+0000 7f366f0c3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-03T17:14:27.388 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:27.386+0000 7fad83fff700 1 -- 172.21.15.67:0/4160988873 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mgrmap(e 14) ==== 99937+0+0 (secure 0 0 0) 0x7fad8c009d10 con 0x7fad9c103ff0 2024-05-03T17:14:27.558 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:27.393+0000 7f366f0c3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-03T17:14:27.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:27 smithi070 ceph-mon[33250]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:27.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:27 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:27.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:27 smithi070 ceph-mon[33250]: Standby manager daemon z started 2024-05-03T17:14:27.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:27 smithi070 ceph-mon[33250]: from='mgr.? 172.21.15.82:0/4203369562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-03T17:14:27.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:27 smithi070 ceph-mon[33250]: from='mgr.? 172.21.15.82:0/4203369562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T17:14:27.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:27 smithi070 ceph-mon[33250]: from='mgr.? 172.21.15.82:0/4203369562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-03T17:14:27.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:27 smithi070 ceph-mon[33250]: from='mgr.? 172.21.15.82:0/4203369562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T17:14:27.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:27 smithi067 ceph-mon[29731]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:27.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:27 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:27.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:27 smithi067 ceph-mon[29731]: Standby manager daemon z started 2024-05-03T17:14:27.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:27 smithi067 ceph-mon[29731]: from='mgr.? 172.21.15.82:0/4203369562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-03T17:14:27.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:27 smithi067 ceph-mon[29731]: from='mgr.? 172.21.15.82:0/4203369562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T17:14:27.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:27 smithi067 ceph-mon[29731]: from='mgr.? 172.21.15.82:0/4203369562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-03T17:14:27.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:27 smithi067 ceph-mon[29731]: from='mgr.? 172.21.15.82:0/4203369562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T17:14:27.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:27 smithi082 ceph-mon[33535]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:27.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:27 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:27.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:27 smithi082 ceph-mon[33535]: Standby manager daemon z started 2024-05-03T17:14:27.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:27 smithi082 ceph-mon[33535]: from='mgr.? 172.21.15.82:0/4203369562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-03T17:14:27.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:27 smithi082 ceph-mon[33535]: from='mgr.? 172.21.15.82:0/4203369562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T17:14:27.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:27 smithi082 ceph-mon[33535]: from='mgr.? 172.21.15.82:0/4203369562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-03T17:14:27.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:27 smithi082 ceph-mon[33535]: from='mgr.? 172.21.15.82:0/4203369562' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T17:14:28.058 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:27.634+0000 7f366f0c3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-03T17:14:28.058 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:27.766+0000 7f366f0c3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-03T17:14:28.390 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:28 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:28.070+0000 7f366f0c3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-03T17:14:28.391 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:28 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:28.183+0000 7f366f0c3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-03T17:14:28.391 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:28 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:28.274+0000 7f366f0c3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-03T17:14:28.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:28 smithi067 ceph-mon[29731]: mgrmap e14: x(active, since 81s), standbys: z 2024-05-03T17:14:28.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-03T17:14:28.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:28 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3880036288' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80962eb1-a506-49bb-a528-dee035eb4322"}]: dispatch 2024-05-03T17:14:28.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:28 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3880036288' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80962eb1-a506-49bb-a528-dee035eb4322"}]': finished 2024-05-03T17:14:28.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:28 smithi067 ceph-mon[29731]: osdmap e5: 1 total, 0 up, 1 in 2024-05-03T17:14:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:28.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:28 smithi082 ceph-mon[33535]: mgrmap e14: x(active, since 81s), standbys: z 2024-05-03T17:14:28.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-03T17:14:28.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:28 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3880036288' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80962eb1-a506-49bb-a528-dee035eb4322"}]: dispatch 2024-05-03T17:14:28.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:28 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3880036288' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80962eb1-a506-49bb-a528-dee035eb4322"}]': finished 2024-05-03T17:14:28.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:28 smithi082 ceph-mon[33535]: osdmap e5: 1 total, 0 up, 1 in 2024-05-03T17:14:28.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:28.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:28 smithi070 ceph-mon[33250]: mgrmap e14: x(active, since 81s), standbys: z 2024-05-03T17:14:28.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-03T17:14:28.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:28 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3880036288' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "80962eb1-a506-49bb-a528-dee035eb4322"}]: dispatch 2024-05-03T17:14:28.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:28 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3880036288' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "80962eb1-a506-49bb-a528-dee035eb4322"}]': finished 2024-05-03T17:14:28.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:28 smithi070 ceph-mon[33250]: osdmap e5: 1 total, 0 up, 1 in 2024-05-03T17:14:28.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:29.308 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:28 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:28.880+0000 7f366f0c3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-03T17:14:29.308 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:14:29 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y[34646]: 2024-05-03T17:14:29.099+0000 7f366f0c3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-03T17:14:29.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:29 smithi067 ceph-mon[29731]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:29.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:29 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1564019103' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:14:29.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:29 smithi067 ceph-mon[29731]: Standby manager daemon y started 2024-05-03T17:14:29.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:29 smithi067 ceph-mon[29731]: from='mgr.? 172.21.15.70:0/2191392360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-03T17:14:29.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:29 smithi067 ceph-mon[29731]: from='mgr.? 172.21.15.70:0/2191392360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T17:14:29.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:29 smithi067 ceph-mon[29731]: from='mgr.? 172.21.15.70:0/2191392360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-03T17:14:29.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:29 smithi067 ceph-mon[29731]: from='mgr.? 172.21.15.70:0/2191392360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T17:14:29.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:29 smithi082 ceph-mon[33535]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:29.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:29 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1564019103' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:14:29.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:29 smithi082 ceph-mon[33535]: Standby manager daemon y started 2024-05-03T17:14:29.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:29 smithi082 ceph-mon[33535]: from='mgr.? 172.21.15.70:0/2191392360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-03T17:14:29.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:29 smithi082 ceph-mon[33535]: from='mgr.? 172.21.15.70:0/2191392360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T17:14:29.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:29 smithi082 ceph-mon[33535]: from='mgr.? 172.21.15.70:0/2191392360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-03T17:14:29.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:29 smithi082 ceph-mon[33535]: from='mgr.? 172.21.15.70:0/2191392360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T17:14:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:29 smithi070 ceph-mon[33250]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:29 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1564019103' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:14:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:29 smithi070 ceph-mon[33250]: Standby manager daemon y started 2024-05-03T17:14:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:29 smithi070 ceph-mon[33250]: from='mgr.? 172.21.15.70:0/2191392360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-03T17:14:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:29 smithi070 ceph-mon[33250]: from='mgr.? 172.21.15.70:0/2191392360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-03T17:14:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:29 smithi070 ceph-mon[33250]: from='mgr.? 172.21.15.70:0/2191392360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-03T17:14:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:29 smithi070 ceph-mon[33250]: from='mgr.? 172.21.15.70:0/2191392360' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-03T17:14:29.820 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:29.819+0000 7fad83fff700 1 -- 172.21.15.67:0/4160988873 <== mon.2 v2:172.21.15.70:3300/0 8 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7fad8c02acc0 con 0x7fad9c103ff0 2024-05-03T17:14:31.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:30 smithi082 ceph-mon[33535]: mgrmap e15: x(active, since 83s), standbys: y, z 2024-05-03T17:14:31.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:30 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-03T17:14:31.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:30 smithi067 ceph-mon[29731]: mgrmap e15: x(active, since 83s), standbys: y, z 2024-05-03T17:14:31.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:30 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-03T17:14:31.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:30 smithi070 ceph-mon[33250]: mgrmap e15: x(active, since 83s), standbys: y, z 2024-05-03T17:14:31.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:30 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-03T17:14:32.094 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:31 smithi067 ceph-mon[29731]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:32.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:31 smithi082 ceph-mon[33535]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:32.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:31 smithi070 ceph-mon[33250]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:33.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:33 smithi067 ceph-mon[29731]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:34.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:33 smithi082 ceph-mon[33535]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:34.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:33 smithi070 ceph-mon[33250]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:36.131 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:35 smithi067 ceph-mon[29731]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:36.131 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:35 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-03T17:14:36.131 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:35 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:36.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:35 smithi082 ceph-mon[33535]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:36.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:35 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-03T17:14:36.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:35 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:36.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:35 smithi070 ceph-mon[33250]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:36.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:35 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-03T17:14:36.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:35 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:37.054 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:36 smithi067 ceph-mon[29731]: Deploying daemon osd.0 on smithi067 2024-05-03T17:14:37.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:36 smithi082 ceph-mon[33535]: Deploying daemon osd.0 on smithi067 2024-05-03T17:14:37.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:36 smithi070 ceph-mon[33250]: Deploying daemon osd.0 on smithi067 2024-05-03T17:14:38.082 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:37 smithi067 ceph-mon[29731]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:38.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:37 smithi082 ceph-mon[33535]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:38.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:37 smithi070 ceph-mon[33250]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:40.133 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:39 smithi067 ceph-mon[29731]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:40.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:39 smithi082 ceph-mon[33535]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:40.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:39 smithi070 ceph-mon[33250]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:42.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:41 smithi082 ceph-mon[33535]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:42.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:41 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:42.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:41 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:42.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:41 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:42.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:41 smithi070 ceph-mon[33250]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:42.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:41 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:42.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:41 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:42.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:41 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:42.185 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:41 smithi067 ceph-mon[29731]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:42.185 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:41 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:42.185 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:41 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:42.185 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:41 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:44.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:43 smithi082 ceph-mon[33535]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:44.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:44.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:44.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:44.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:44.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:44.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:43 smithi070 ceph-mon[33250]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:44.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:44.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:44.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:44.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:44.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:44.135 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:43 smithi067 ceph-mon[29731]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:44.136 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:44.136 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:44.136 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:44.136 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:44.136 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:44.543 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 0 on host 'smithi067' 2024-05-03T17:14:44.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.542+0000 7fad83fff700 1 -- 172.21.15.67:0/4160988873 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fad9c0638e0 con 0x7fad8403e750 2024-05-03T17:14:44.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.545+0000 7fada1597700 1 -- 172.21.15.67:0/4160988873 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fad8403e750 msgr2=0x7fad84040c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:44.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.545+0000 7fada1597700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fad8403e750 0x7fad84040c10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fad90011490 tx=0x7fad90009260 comp rx=0 tx=0).stop 2024-05-03T17:14:44.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.545+0000 7fada1597700 1 -- 172.21.15.67:0/4160988873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9c103ff0 msgr2=0x7fad9c19b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:44.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.545+0000 7fada1597700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9c103ff0 0x7fad9c19b920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fad8c00b650 tx=0x7fad8c0047b0 comp rx=0 tx=0).stop 2024-05-03T17:14:44.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.545+0000 7fada1597700 1 -- 172.21.15.67:0/4160988873 shutdown_connections 2024-05-03T17:14:44.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.545+0000 7fada1597700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fad8403e750 0x7fad84040c10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:44.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.545+0000 7fada1597700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fad9c109250 0x7fad9c1a2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:44.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.545+0000 7fada1597700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fad9c103ff0 0x7fad9c19b920 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:44.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.545+0000 7fada1597700 1 --2- 172.21.15.67:0/4160988873 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fad9c106920 0x7fad9c19be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:44.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.545+0000 7fada1597700 1 -- 172.21.15.67:0/4160988873 >> 172.21.15.67:0/4160988873 conn(0x7fad9c0fd9a0 msgr2=0x7fad9c1037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:14:44.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.545+0000 7fada1597700 1 -- 172.21.15.67:0/4160988873 shutdown_connections 2024-05-03T17:14:44.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:14:44.545+0000 7fada1597700 1 -- 172.21.15.67:0/4160988873 wait complete. 2024-05-03T17:14:45.284 DEBUG:teuthology.orchestra.run.smithi067:osd.0> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.0.service 2024-05-03T17:14:45.286 INFO:tasks.cephadm:Deploying osd.1 on smithi070 with /dev/vg_nvme/lv_4... 2024-05-03T17:14:45.287 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-03T17:14:45.513 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:45 smithi067 ceph-mon[29731]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:45.513 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:45 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:45.513 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:45 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:45.513 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:45 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:45.513 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:45 smithi067 ceph-mon[29731]: from='osd.0 [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T17:14:45.513 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:45 smithi067 ceph-mon[29731]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T17:14:45.531 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T17:14:45.642 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:45 smithi070 ceph-mon[33250]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:45.642 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:45 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:45.642 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:45 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:45.642 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:45 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:45.642 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:45 smithi070 ceph-mon[33250]: from='osd.0 [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T17:14:45.642 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:45 smithi070 ceph-mon[33250]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T17:14:45.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:45 smithi082 ceph-mon[33535]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:45.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:45 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:45.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:45 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:45.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:45 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:14:45.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:45 smithi082 ceph-mon[33535]: from='osd.0 [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T17:14:45.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:45 smithi082 ceph-mon[33535]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-03T17:14:46.755 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:46 smithi067 ceph-mon[29731]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-03T17:14:46.756 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:46 smithi067 ceph-mon[29731]: osdmap e6: 1 total, 0 up, 1 in 2024-05-03T17:14:46.756 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:46.756 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:46 smithi067 ceph-mon[29731]: from='osd.0 [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:14:46.756 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:46 smithi067 ceph-mon[29731]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:14:46.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:46 smithi082 ceph-mon[33535]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-03T17:14:46.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:46 smithi082 ceph-mon[33535]: osdmap e6: 1 total, 0 up, 1 in 2024-05-03T17:14:46.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:46.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:46 smithi082 ceph-mon[33535]: from='osd.0 [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:14:46.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:46 smithi082 ceph-mon[33535]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:14:46.791 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:46 smithi070 ceph-mon[33250]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-03T17:14:46.792 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:46 smithi070 ceph-mon[33250]: osdmap e6: 1 total, 0 up, 1 in 2024-05-03T17:14:46.792 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:46.792 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:46 smithi070 ceph-mon[33250]: from='osd.0 [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:14:46.792 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:46 smithi070 ceph-mon[33250]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:14:47.101 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:14:46 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0[44787]: 2024-05-03T17:14:46.997+0000 7f46f86e9700 -1 osd.0 0 waiting for initial osdmap 2024-05-03T17:14:47.101 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:14:47 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0[44787]: 2024-05-03T17:14:47.000+0000 7f46f04d4700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T17:14:47.341 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-03T17:14:47.341 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-03T17:14:47.341 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-05-03T17:14:47.341 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-05-03T17:14:47.341 INFO:teuthology.orchestra.run.smithi070.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00843103 s, 1.2 GB/s 2024-05-03T17:14:47.341 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-05-03T17:14:47.653 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:47 smithi070 ceph-mon[33250]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:47.653 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:47 smithi070 ceph-mon[33250]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T17:14:47.653 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:47 smithi070 ceph-mon[33250]: osdmap e7: 1 total, 0 up, 1 in 2024-05-03T17:14:47.653 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:47 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:47.654 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:47 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:47.654 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:47 smithi070 ceph-mon[33250]: from='osd.0 ' entity='osd.0' 2024-05-03T17:14:47.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:47 smithi082 ceph-mon[33535]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:47.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:47 smithi082 ceph-mon[33535]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T17:14:47.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:47 smithi082 ceph-mon[33535]: osdmap e7: 1 total, 0 up, 1 in 2024-05-03T17:14:47.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:47 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:47.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:47 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:47.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:47 smithi082 ceph-mon[33535]: from='osd.0 ' entity='osd.0' 2024-05-03T17:14:47.938 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_4 2024-05-03T17:14:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:47 smithi067 ceph-mon[29731]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:47 smithi067 ceph-mon[29731]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T17:14:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:47 smithi067 ceph-mon[29731]: osdmap e7: 1 total, 0 up, 1 in 2024-05-03T17:14:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:47 smithi067 ceph-mon[29731]: from='osd.0 ' entity='osd.0' 2024-05-03T17:14:48.193 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T17:14:48.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:48 smithi070 ceph-mon[33250]: purged_snaps scrub starts 2024-05-03T17:14:48.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:48 smithi070 ceph-mon[33250]: purged_snaps scrub ok 2024-05-03T17:14:48.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:48 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:48.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:48 smithi070 ceph-mon[33250]: osd.0 [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741] boot 2024-05-03T17:14:48.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:48 smithi070 ceph-mon[33250]: osdmap e8: 1 total, 1 up, 1 in 2024-05-03T17:14:48.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:48 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:48.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:48 smithi067 ceph-mon[29731]: purged_snaps scrub starts 2024-05-03T17:14:48.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:48 smithi067 ceph-mon[29731]: purged_snaps scrub ok 2024-05-03T17:14:48.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:48 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:48.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:48 smithi067 ceph-mon[29731]: osd.0 [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741] boot 2024-05-03T17:14:48.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:48 smithi067 ceph-mon[29731]: osdmap e8: 1 total, 1 up, 1 in 2024-05-03T17:14:48.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:48 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:48.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:48 smithi082 ceph-mon[33535]: purged_snaps scrub starts 2024-05-03T17:14:48.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:48 smithi082 ceph-mon[33535]: purged_snaps scrub ok 2024-05-03T17:14:48.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:48 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:48.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:48 smithi082 ceph-mon[33535]: osd.0 [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741] boot 2024-05-03T17:14:48.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:48 smithi082 ceph-mon[33535]: osdmap e8: 1 total, 1 up, 1 in 2024-05-03T17:14:48.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:48 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-03T17:14:49.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.639+0000 7fd80fb62700 1 -- 172.21.15.70:0/1012035923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd808103fc0 msgr2=0x7fd80810a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:49.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.639+0000 7fd80fb62700 1 --2- 172.21.15.70:0/1012035923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd808103fc0 0x7fd80810a870 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd80000b0d0 tx=0x7fd80000b3e0 comp rx=0 tx=0).stop 2024-05-03T17:14:49.644 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.639+0000 7fd80fb62700 1 -- 172.21.15.70:0/1012035923 shutdown_connections 2024-05-03T17:14:49.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.639+0000 7fd80fb62700 1 --2- 172.21.15.70:0/1012035923 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd808103fc0 0x7fd80810a870 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:49.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.639+0000 7fd80fb62700 1 --2- 172.21.15.70:0/1012035923 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd808102420 0x7fd808102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:49.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.639+0000 7fd80fb62700 1 --2- 172.21.15.70:0/1012035923 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd808103620 0x7fd808103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:49.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.639+0000 7fd80fb62700 1 -- 172.21.15.70:0/1012035923 >> 172.21.15.70:0/1012035923 conn(0x7fd8080fda00 msgr2=0x7fd8080ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:14:49.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.639+0000 7fd80fb62700 1 -- 172.21.15.70:0/1012035923 shutdown_connections 2024-05-03T17:14:49.645 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.639+0000 7fd80fb62700 1 -- 172.21.15.70:0/1012035923 wait complete. 2024-05-03T17:14:49.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.640+0000 7fd80fb62700 1 Processor -- start 2024-05-03T17:14:49.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.640+0000 7fd80fb62700 1 -- start start 2024-05-03T17:14:49.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.640+0000 7fd80fb62700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd808102420 0x7fd808198650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:49.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.640+0000 7fd80fb62700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd808103620 0x7fd808198b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:49.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.640+0000 7fd80fb62700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd808103fc0 0x7fd80819fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:49.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.640+0000 7fd80fb62700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd80810dfb0 con 0x7fd808103620 2024-05-03T17:14:49.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.640+0000 7fd80fb62700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd80810de30 con 0x7fd808103fc0 2024-05-03T17:14:49.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.640+0000 7fd80fb62700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd80810e130 con 0x7fd808102420 2024-05-03T17:14:49.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.640+0000 7fd80e0ff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd808103fc0 0x7fd80819fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:49.646 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.640+0000 7fd80e0ff700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd808103fc0 0x7fd80819fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:52602/0 (socket says 172.21.15.70:52602) 2024-05-03T17:14:49.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.640+0000 7fd80e0ff700 1 -- 172.21.15.70:0/1822126247 learned_addr learned my addr 172.21.15.70:0/1822126247 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-05-03T17:14:49.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.640+0000 7fd80d0fd700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd808103620 0x7fd808198b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:49.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.641+0000 7fd80d8fe700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd808102420 0x7fd808198650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:49.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.641+0000 7fd80e0ff700 1 -- 172.21.15.70:0/1822126247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd808102420 msgr2=0x7fd808198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:49.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.641+0000 7fd80e0ff700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd808102420 0x7fd808198650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:49.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.641+0000 7fd80e0ff700 1 -- 172.21.15.70:0/1822126247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd808103620 msgr2=0x7fd808198b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:14:49.647 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.641+0000 7fd80e0ff700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd808103620 0x7fd808198b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:14:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.641+0000 7fd80e0ff700 1 -- 172.21.15.70:0/1822126247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd80806bc10 con 0x7fd808103fc0 2024-05-03T17:14:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.641+0000 7fd80e0ff700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd808103fc0 0x7fd80819fc10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd800009fd0 tx=0x7fd800004e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:14:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.641+0000 7fd7feffd700 1 -- 172.21.15.70:0/1822126247 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80001f030 con 0x7fd808103fc0 2024-05-03T17:14:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.641+0000 7fd80fb62700 1 -- 172.21.15.70:0/1822126247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd80806bea0 con 0x7fd808103fc0 2024-05-03T17:14:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.641+0000 7fd7feffd700 1 -- 172.21.15.70:0/1822126247 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd800007a80 con 0x7fd808103fc0 2024-05-03T17:14:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.641+0000 7fd7feffd700 1 -- 172.21.15.70:0/1822126247 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd800004810 con 0x7fd808103fc0 2024-05-03T17:14:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.641+0000 7fd80fb62700 1 -- 172.21.15.70:0/1822126247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd80806c400 con 0x7fd808103fc0 2024-05-03T17:14:49.648 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.642+0000 7fd80fb62700 1 -- 172.21.15.70:0/1822126247 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd80806a190 con 0x7fd808103fc0 2024-05-03T17:14:49.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.644+0000 7fd7feffd700 1 -- 172.21.15.70:0/1822126247 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7fd800003a00 con 0x7fd808103fc0 2024-05-03T17:14:49.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.644+0000 7fd7feffd700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd7f40b27f0 0x7fd7f40b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:14:49.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.644+0000 7fd7feffd700 1 -- 172.21.15.70:0/1822126247 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fd8000e1680 con 0x7fd808103fc0 2024-05-03T17:14:49.649 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.645+0000 7fd80d8fe700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd7f40b27f0 0x7fd7f40b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:14:49.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.645+0000 7fd80d8fe700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd7f40b27f0 0x7fd7f40b4cb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd804006fd0 tx=0x7fd804008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:14:49.650 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.648+0000 7fd7feffd700 1 -- 172.21.15.70:0/1822126247 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fd8000aaea0 con 0x7fd808103fc0 2024-05-03T17:14:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:49 smithi082 ceph-mon[33535]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:49 smithi082 ceph-mon[33535]: Detected new or changed devices on smithi067 2024-05-03T17:14:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:14:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:49 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi067 to 17088M 2024-05-03T17:14:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:49.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:49 smithi070 ceph-mon[33250]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:49.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:49 smithi070 ceph-mon[33250]: Detected new or changed devices on smithi067 2024-05-03T17:14:49.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:49.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:49.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:14:49.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:49 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi067 to 17088M 2024-05-03T17:14:49.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:49.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:49.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:49.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:49.861 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:14:49.859+0000 7fd80fb62700 1 -- 172.21.15.70:0/1822126247 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd80806c660 con 0x7fd7f40b27f0 2024-05-03T17:14:49.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:49 smithi067 ceph-mon[29731]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-03T17:14:49.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:49 smithi067 ceph-mon[29731]: Detected new or changed devices on smithi067 2024-05-03T17:14:49.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:49.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:49.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:14:49.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:49 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi067 to 17088M 2024-05-03T17:14:49.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:49.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:49.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:14:49.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:14:50.707 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:50 smithi070 ceph-mon[33250]: osdmap e9: 1 total, 1 up, 1 in 2024-05-03T17:14:50.708 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:50 smithi070 ceph-mon[33250]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:14:50.708 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:14:50.708 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:14:50.708 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:50.854 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:50 smithi067 ceph-mon[29731]: osdmap e9: 1 total, 1 up, 1 in 2024-05-03T17:14:50.854 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:50 smithi067 ceph-mon[29731]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:14:50.854 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:14:50.854 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:14:50.854 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:51.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:50 smithi082 ceph-mon[33535]: osdmap e9: 1 total, 1 up, 1 in 2024-05-03T17:14:51.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:50 smithi082 ceph-mon[33535]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:14:51.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:14:51.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:14:51.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:51.684 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:51 smithi070 ceph-mon[33250]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:51.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:51 smithi067 ceph-mon[29731]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:52.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:51 smithi082 ceph-mon[33535]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:52.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:52 smithi070 ceph-mon[33250]: from='client.? 172.21.15.70:0/343501338' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7721d5f-4373-4cb2-bb57-bc368d6bb818"}]: dispatch 2024-05-03T17:14:52.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:52 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7721d5f-4373-4cb2-bb57-bc368d6bb818"}]: dispatch 2024-05-03T17:14:52.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:52 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7721d5f-4373-4cb2-bb57-bc368d6bb818"}]': finished 2024-05-03T17:14:52.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:52 smithi070 ceph-mon[33250]: osdmap e10: 2 total, 1 up, 2 in 2024-05-03T17:14:52.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:52 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:14:52.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:52 smithi067 ceph-mon[29731]: from='client.? 172.21.15.70:0/343501338' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7721d5f-4373-4cb2-bb57-bc368d6bb818"}]: dispatch 2024-05-03T17:14:52.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:52 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7721d5f-4373-4cb2-bb57-bc368d6bb818"}]: dispatch 2024-05-03T17:14:52.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:52 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7721d5f-4373-4cb2-bb57-bc368d6bb818"}]': finished 2024-05-03T17:14:52.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:52 smithi067 ceph-mon[29731]: osdmap e10: 2 total, 1 up, 2 in 2024-05-03T17:14:52.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:52 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:14:53.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:52 smithi082 ceph-mon[33535]: from='client.? 172.21.15.70:0/343501338' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7721d5f-4373-4cb2-bb57-bc368d6bb818"}]: dispatch 2024-05-03T17:14:53.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:52 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d7721d5f-4373-4cb2-bb57-bc368d6bb818"}]: dispatch 2024-05-03T17:14:53.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:52 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d7721d5f-4373-4cb2-bb57-bc368d6bb818"}]': finished 2024-05-03T17:14:53.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:52 smithi082 ceph-mon[33535]: osdmap e10: 2 total, 1 up, 2 in 2024-05-03T17:14:53.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:52 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:14:53.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:53 smithi070 ceph-mon[33250]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:53.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:53 smithi070 ceph-mon[33250]: from='client.? 172.21.15.70:0/977725561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:14:53.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:53 smithi067 ceph-mon[29731]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:53.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:53 smithi067 ceph-mon[29731]: from='client.? 172.21.15.70:0/977725561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:14:54.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:53 smithi082 ceph-mon[33535]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:54.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:53 smithi082 ceph-mon[33535]: from='client.? 172.21.15.70:0/977725561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:14:55.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:55 smithi070 ceph-mon[33250]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:55.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:55 smithi067 ceph-mon[29731]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:56.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:55 smithi082 ceph-mon[33535]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:57.606 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:57 smithi070 ceph-mon[33250]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:57.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:57 smithi082 ceph-mon[33535]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:57.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:57 smithi067 ceph-mon[29731]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:59.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:59 smithi070 ceph-mon[33250]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:59.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-03T17:14:59.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:14:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:14:59.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:59 smithi067 ceph-mon[29731]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:14:59.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-03T17:14:59.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:14:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:00.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:59 smithi082 ceph-mon[33535]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:00.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-03T17:15:00.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:14:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:00.712 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:00 smithi070 ceph-mon[33250]: Deploying daemon osd.1 on smithi070 2024-05-03T17:15:00.906 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:00 smithi067 ceph-mon[29731]: Deploying daemon osd.1 on smithi070 2024-05-03T17:15:01.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:00 smithi082 ceph-mon[33535]: Deploying daemon osd.1 on smithi070 2024-05-03T17:15:01.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:01 smithi070 ceph-mon[33250]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:01.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:01 smithi067 ceph-mon[29731]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:02.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:01 smithi082 ceph-mon[33535]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:03.890 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:03 smithi070 ceph-mon[33250]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:03.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:03 smithi067 ceph-mon[29731]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:04.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:03 smithi082 ceph-mon[33535]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:05.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:05 smithi067 ceph-mon[29731]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:05.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:05.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:05.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:05.994 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:05 smithi070 ceph-mon[33250]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:05.994 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:05.994 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:05.994 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:06.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:05 smithi082 ceph-mon[33535]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:06.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:06.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:06.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:07.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:07 smithi070 ceph-mon[33250]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:07.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:07.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:07.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:07.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:07.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:07.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:07 smithi067 ceph-mon[29731]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:07.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:07.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:07.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:07.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:07.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:07 smithi082 ceph-mon[33535]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:08.128 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 1 on host 'smithi070' 2024-05-03T17:15:08.128 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.127+0000 7fd7feffd700 1 -- 172.21.15.70:0/1822126247 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd80806c660 con 0x7fd7f40b27f0 2024-05-03T17:15:08.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.129+0000 7fd80fb62700 1 -- 172.21.15.70:0/1822126247 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd7f40b27f0 msgr2=0x7fd7f40b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:15:08.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.129+0000 7fd80fb62700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd7f40b27f0 0x7fd7f40b4cb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd804006fd0 tx=0x7fd804008040 comp rx=0 tx=0).stop 2024-05-03T17:15:08.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.129+0000 7fd80fb62700 1 -- 172.21.15.70:0/1822126247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd808103fc0 msgr2=0x7fd80819fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:15:08.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.129+0000 7fd80fb62700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd808103fc0 0x7fd80819fc10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd800009fd0 tx=0x7fd800004e40 comp rx=0 tx=0).stop 2024-05-03T17:15:08.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.129+0000 7fd80fb62700 1 -- 172.21.15.70:0/1822126247 shutdown_connections 2024-05-03T17:15:08.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.129+0000 7fd80fb62700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd7f40b27f0 0x7fd7f40b4cb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:08.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.129+0000 7fd80fb62700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd808103fc0 0x7fd80819fc10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:08.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.129+0000 7fd80fb62700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd808102420 0x7fd808198650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:08.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.129+0000 7fd80fb62700 1 --2- 172.21.15.70:0/1822126247 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd808103620 0x7fd808198b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:08.130 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.129+0000 7fd80fb62700 1 -- 172.21.15.70:0/1822126247 >> 172.21.15.70:0/1822126247 conn(0x7fd8080fda00 msgr2=0x7fd808106a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:15:08.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.129+0000 7fd80fb62700 1 -- 172.21.15.70:0/1822126247 shutdown_connections 2024-05-03T17:15:08.131 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:15:08.129+0000 7fd80fb62700 1 -- 172.21.15.70:0/1822126247 wait complete. 2024-05-03T17:15:08.789 DEBUG:teuthology.orchestra.run.smithi070:osd.1> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.1.service 2024-05-03T17:15:08.791 INFO:tasks.cephadm:Deploying osd.2 on smithi082 with /dev/vg_nvme/lv_4... 2024-05-03T17:15:08.791 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-03T17:15:09.036 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:15:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:09 smithi070 ceph-mon[33250]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:09 smithi070 ceph-mon[33250]: from='osd.1 [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T17:15:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:09 smithi070 ceph-mon[33250]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T17:15:09.380 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:09.380 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:09.380 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:09.380 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:09 smithi082 ceph-mon[33535]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:09.380 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:09 smithi082 ceph-mon[33535]: from='osd.1 [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T17:15:09.380 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:09 smithi082 ceph-mon[33535]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T17:15:09.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:09.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:09.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:09.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:09 smithi067 ceph-mon[29731]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:09.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:09 smithi067 ceph-mon[29731]: from='osd.1 [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T17:15:09.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:09 smithi067 ceph-mon[29731]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-03T17:15:10.441 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:10 smithi082 ceph-mon[33535]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-03T17:15:10.441 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:10 smithi082 ceph-mon[33535]: from='osd.1 [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:15:10.441 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:10 smithi082 ceph-mon[33535]: osdmap e11: 2 total, 1 up, 2 in 2024-05-03T17:15:10.441 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:10.441 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:10 smithi082 ceph-mon[33535]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:15:10.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:10 smithi067 ceph-mon[29731]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-03T17:15:10.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:10 smithi067 ceph-mon[29731]: from='osd.1 [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:15:10.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:10 smithi067 ceph-mon[29731]: osdmap e11: 2 total, 1 up, 2 in 2024-05-03T17:15:10.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:10.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:10 smithi067 ceph-mon[29731]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:15:10.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:10 smithi070 ceph-mon[33250]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-03T17:15:10.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:10 smithi070 ceph-mon[33250]: from='osd.1 [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:15:10.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:10 smithi070 ceph-mon[33250]: osdmap e11: 2 total, 1 up, 2 in 2024-05-03T17:15:10.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:10.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:10 smithi070 ceph-mon[33250]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:15:10.559 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:15:10 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1[37861]: 2024-05-03T17:15:10.139+0000 7f870f9e2700 -1 osd.1 0 waiting for initial osdmap 2024-05-03T17:15:10.559 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:15:10 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1[37861]: 2024-05-03T17:15:10.143+0000 7f870b7d5700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T17:15:11.000 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-03T17:15:11.000 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-03T17:15:11.000 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-05-03T17:15:11.000 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-05-03T17:15:11.000 INFO:teuthology.orchestra.run.smithi082.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123754 s, 847 MB/s 2024-05-03T17:15:11.000 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-05-03T17:15:11.229 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:11 smithi070 ceph-mon[33250]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-05-03T17:15:11.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:11 smithi070 ceph-mon[33250]: osdmap e12: 2 total, 1 up, 2 in 2024-05-03T17:15:11.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:11 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:11.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:11 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:11.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:11 smithi070 ceph-mon[33250]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:11.230 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:11 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:11.298 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:11 smithi082 ceph-mon[33535]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-05-03T17:15:11.298 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:11 smithi082 ceph-mon[33535]: osdmap e12: 2 total, 1 up, 2 in 2024-05-03T17:15:11.298 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:11 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:11.298 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:11 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:11.298 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:11 smithi082 ceph-mon[33535]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:11.298 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:11 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:11.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:11 smithi067 ceph-mon[29731]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-05-03T17:15:11.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:11 smithi067 ceph-mon[29731]: osdmap e12: 2 total, 1 up, 2 in 2024-05-03T17:15:11.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:11 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:11.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:11 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:11.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:11 smithi067 ceph-mon[29731]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-03T17:15:11.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:11 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:11.594 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_4 2024-05-03T17:15:11.834 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:15:12.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: purged_snaps scrub starts 2024-05-03T17:15:12.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: purged_snaps scrub ok 2024-05-03T17:15:12.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: osd.1 [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176] boot 2024-05-03T17:15:12.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: osdmap e13: 2 total, 2 up, 2 in 2024-05-03T17:15:12.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:12.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: Detected new or changed devices on smithi070 2024-05-03T17:15:12.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:12.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:12.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:15:12.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi070 to 17088M 2024-05-03T17:15:12.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:12.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:12.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:12.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:12.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: purged_snaps scrub starts 2024-05-03T17:15:12.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: purged_snaps scrub ok 2024-05-03T17:15:12.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: osd.1 [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176] boot 2024-05-03T17:15:12.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: osdmap e13: 2 total, 2 up, 2 in 2024-05-03T17:15:12.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:12.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: Detected new or changed devices on smithi070 2024-05-03T17:15:12.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:12.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:12.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:15:12.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi070 to 17088M 2024-05-03T17:15:12.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:12.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:12.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:12.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:12.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: purged_snaps scrub starts 2024-05-03T17:15:12.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: purged_snaps scrub ok 2024-05-03T17:15:12.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: osd.1 [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176] boot 2024-05-03T17:15:12.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: osdmap e13: 2 total, 2 up, 2 in 2024-05-03T17:15:12.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-03T17:15:12.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: Detected new or changed devices on smithi070 2024-05-03T17:15:12.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:12.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:12.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:15:12.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi070 to 17088M 2024-05-03T17:15:12.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:12.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:12.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:12.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:13.372 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.369+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/866555638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c58103fe0 msgr2=0x7f1c5810a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:15:13.372 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.369+0000 7f1c5ffc1700 1 --2- 172.21.15.82:0/866555638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c58103fe0 0x7f1c5810a890 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1c5000b0a0 tx=0x7f1c5000b3b0 comp rx=0 tx=0).stop 2024-05-03T17:15:13.372 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.369+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/866555638 shutdown_connections 2024-05-03T17:15:13.372 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.369+0000 7f1c5ffc1700 1 --2- 172.21.15.82:0/866555638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c58103fe0 0x7f1c5810a890 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:13.372 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.369+0000 7f1c5ffc1700 1 --2- 172.21.15.82:0/866555638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c58102440 0x7f1c58102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:13.372 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.369+0000 7f1c5ffc1700 1 --2- 172.21.15.82:0/866555638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c58103640 0x7f1c58103aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:13.372 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.369+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/866555638 >> 172.21.15.82:0/866555638 conn(0x7f1c580fd9c0 msgr2=0x7f1c580ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:15:13.372 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.369+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/866555638 shutdown_connections 2024-05-03T17:15:13.372 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.369+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/866555638 wait complete. 2024-05-03T17:15:13.372 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.369+0000 7f1c5ffc1700 1 Processor -- start 2024-05-03T17:15:13.372 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.369+0000 7f1c5ffc1700 1 -- start start 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5ffc1700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c58102440 0x7f1c581063d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5ffc1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c58103640 0x7f1c58104a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5ffc1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c58103fe0 0x7f1c58104f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5ffc1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5810df30 con 0x7f1c58103fe0 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5ffc1700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5810ddb0 con 0x7f1c58103640 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5ffc1700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1c5810e0b0 con 0x7f1c58102440 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5dd5d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c58102440 0x7f1c581063d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5dd5d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c58102440 0x7f1c581063d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:38378/0 (socket says 172.21.15.82:38378) 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5dd5d700 1 -- 172.21.15.82:0/586018155 learned_addr learned my addr 172.21.15.82:0/586018155 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5e55e700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c58103fe0 0x7f1c58104f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5d55c700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c58103640 0x7f1c58104a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5dd5d700 1 -- 172.21.15.82:0/586018155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c58103640 msgr2=0x7f1c58104a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5dd5d700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c58103640 0x7f1c58104a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:13.373 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5dd5d700 1 -- 172.21.15.82:0/586018155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c58103fe0 msgr2=0x7f1c58104f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:15:13.374 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5dd5d700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c58103fe0 0x7f1c58104f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:13.374 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5dd5d700 1 -- 172.21.15.82:0/586018155 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c58105810 con 0x7f1c58102440 2024-05-03T17:15:13.374 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5dd5d700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c58102440 0x7f1c581063d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1c5400c8f0 tx=0x7f1c5400cc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:15:13.374 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c4effd700 1 -- 172.21.15.82:0/586018155 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c54004220 con 0x7f1c58102440 2024-05-03T17:15:13.374 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c4effd700 1 -- 172.21.15.82:0/586018155 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c540043c0 con 0x7f1c58102440 2024-05-03T17:15:13.374 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c4effd700 1 -- 172.21.15.82:0/586018155 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c54003c00 con 0x7f1c58102440 2024-05-03T17:15:13.374 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/586018155 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c581a37f0 con 0x7f1c58102440 2024-05-03T17:15:13.374 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.370+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/586018155 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c581a3cb0 con 0x7f1c58102440 2024-05-03T17:15:13.374 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.371+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/586018155 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c5806a190 con 0x7f1c58102440 2024-05-03T17:15:13.375 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.373+0000 7f1c4effd700 1 -- 172.21.15.82:0/586018155 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7f1c54007c10 con 0x7f1c58102440 2024-05-03T17:15:13.375 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.373+0000 7f1c4effd700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1c440b27f0 0x7f1c440b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:15:13.375 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.374+0000 7f1c4effd700 1 -- 172.21.15.82:0/586018155 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f1c540e0890 con 0x7f1c58102440 2024-05-03T17:15:13.375 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.374+0000 7f1c5d55c700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1c440b27f0 0x7f1c440b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:15:13.375 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.374+0000 7f1c5d55c700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1c440b27f0 0x7f1c440b4cb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1c48005950 tx=0x7f1c4800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:15:13.377 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.377+0000 7f1c4effd700 1 -- 172.21.15.82:0/586018155 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f1c54014030 con 0x7f1c58102440 2024-05-03T17:15:13.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:13 smithi067 ceph-mon[29731]: osdmap e14: 2 total, 2 up, 2 in 2024-05-03T17:15:13.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:13 smithi067 ceph-mon[29731]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:13.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:13 smithi082 ceph-mon[33535]: osdmap e14: 2 total, 2 up, 2 in 2024-05-03T17:15:13.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:13 smithi082 ceph-mon[33535]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:13.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:13 smithi070 ceph-mon[33250]: osdmap e14: 2 total, 2 up, 2 in 2024-05-03T17:15:13.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:13 smithi070 ceph-mon[33250]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:13.589 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:13.589+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/586018155 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1c581a3f70 con 0x7f1c440b27f0 2024-05-03T17:15:14.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:14 smithi067 ceph-mon[29731]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:15:14.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:14 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:15:14.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:14 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:15:14.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:14 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:14.495 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:14 smithi082 ceph-mon[33535]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:15:14.495 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:14 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:15:14.495 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:14 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:15:14.495 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:14 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:14.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:14 smithi070 ceph-mon[33250]: from='client.24151 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:15:14.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:14 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:15:14.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:14 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:15:14.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:14 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:15.378 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:15 smithi082 ceph-mon[33535]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:15.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:15 smithi067 ceph-mon[29731]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:15.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:15 smithi070 ceph-mon[33250]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:16.551 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:16 smithi082 ceph-mon[33535]: from='client.? 172.21.15.82:0/1317504516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9992fd98-821b-40ce-b051-89befbfd87c8"}]: dispatch 2024-05-03T17:15:16.551 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:16 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9992fd98-821b-40ce-b051-89befbfd87c8"}]: dispatch 2024-05-03T17:15:16.551 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:16 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9992fd98-821b-40ce-b051-89befbfd87c8"}]': finished 2024-05-03T17:15:16.551 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:16 smithi082 ceph-mon[33535]: osdmap e15: 3 total, 2 up, 3 in 2024-05-03T17:15:16.552 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:16 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:16.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:16 smithi070 ceph-mon[33250]: from='client.? 172.21.15.82:0/1317504516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9992fd98-821b-40ce-b051-89befbfd87c8"}]: dispatch 2024-05-03T17:15:16.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:16 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9992fd98-821b-40ce-b051-89befbfd87c8"}]: dispatch 2024-05-03T17:15:16.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:16 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9992fd98-821b-40ce-b051-89befbfd87c8"}]': finished 2024-05-03T17:15:16.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:16 smithi070 ceph-mon[33250]: osdmap e15: 3 total, 2 up, 3 in 2024-05-03T17:15:16.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:16 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:16.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:16 smithi067 ceph-mon[29731]: from='client.? 172.21.15.82:0/1317504516' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9992fd98-821b-40ce-b051-89befbfd87c8"}]: dispatch 2024-05-03T17:15:16.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:16 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9992fd98-821b-40ce-b051-89befbfd87c8"}]: dispatch 2024-05-03T17:15:16.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:16 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9992fd98-821b-40ce-b051-89befbfd87c8"}]': finished 2024-05-03T17:15:16.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:16 smithi067 ceph-mon[29731]: osdmap e15: 3 total, 2 up, 3 in 2024-05-03T17:15:16.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:16 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:17.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:17 smithi070 ceph-mon[33250]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:17.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:17 smithi070 ceph-mon[33250]: from='client.? 172.21.15.82:0/687563475' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:15:17.560 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:17 smithi082 ceph-mon[33535]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:17.561 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:17 smithi082 ceph-mon[33535]: from='client.? 172.21.15.82:0/687563475' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:15:17.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:17 smithi067 ceph-mon[29731]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:17.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:17 smithi067 ceph-mon[29731]: from='client.? 172.21.15.82:0/687563475' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:15:19.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:19 smithi067 ceph-mon[29731]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:19.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:19 smithi082 ceph-mon[33535]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:19.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:19 smithi070 ceph-mon[33250]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:21.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:21 smithi067 ceph-mon[29731]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:21.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:21 smithi082 ceph-mon[33535]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:21.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:21 smithi070 ceph-mon[33250]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:23.706 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:23 smithi067 ceph-mon[29731]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:23.706 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-03T17:15:23.706 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:23.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:23 smithi082 ceph-mon[33535]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:23.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-03T17:15:23.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:23.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:23 smithi070 ceph-mon[33250]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:23.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-03T17:15:23.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:24.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:24 smithi082 ceph-mon[33535]: Deploying daemon osd.2 on smithi082 2024-05-03T17:15:24.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:24 smithi067 ceph-mon[29731]: Deploying daemon osd.2 on smithi082 2024-05-03T17:15:24.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:24 smithi070 ceph-mon[33250]: Deploying daemon osd.2 on smithi082 2024-05-03T17:15:25.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:25 smithi067 ceph-mon[29731]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:25.762 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:25 smithi082 ceph-mon[33535]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:25.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:25 smithi070 ceph-mon[33250]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:27.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:27 smithi067 ceph-mon[29731]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:27.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:27 smithi082 ceph-mon[33535]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:27.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:27 smithi070 ceph-mon[33250]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:29.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:29 smithi067 ceph-mon[29731]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:29.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:29 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:29.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:29 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:29.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:29 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:30.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:29 smithi082 ceph-mon[33535]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:30.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:29 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:30.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:29 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:30.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:29 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:30.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:29 smithi070 ceph-mon[33250]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:30.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:29 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:30.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:29 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:30.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:29 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:31 smithi067 ceph-mon[29731]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:31.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:31.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:31.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:31.971 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:31 smithi082 ceph-mon[33535]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:31.971 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:31.971 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:31.971 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:31.971 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:31.971 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:31 smithi070 ceph-mon[33250]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:32.305 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 2 on host 'smithi082' 2024-05-03T17:15:32.305 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.304+0000 7f1c4effd700 1 -- 172.21.15.82:0/586018155 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1c581a3f70 con 0x7f1c440b27f0 2024-05-03T17:15:32.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.307+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/586018155 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1c440b27f0 msgr2=0x7f1c440b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:15:32.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.307+0000 7f1c5ffc1700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1c440b27f0 0x7f1c440b4cb0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1c48005950 tx=0x7f1c4800b410 comp rx=0 tx=0).stop 2024-05-03T17:15:32.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.307+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/586018155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c58102440 msgr2=0x7f1c581063d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:15:32.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.307+0000 7f1c5ffc1700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c58102440 0x7f1c581063d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1c5400c8f0 tx=0x7f1c5400cc00 comp rx=0 tx=0).stop 2024-05-03T17:15:32.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.307+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/586018155 shutdown_connections 2024-05-03T17:15:32.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.307+0000 7f1c5ffc1700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1c440b27f0 0x7f1c440b4cb0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:32.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.307+0000 7f1c5ffc1700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1c58103fe0 0x7f1c58104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:32.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.307+0000 7f1c5ffc1700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1c58102440 0x7f1c581063d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:32.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.307+0000 7f1c5ffc1700 1 --2- 172.21.15.82:0/586018155 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1c58103640 0x7f1c58104a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:32.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.307+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/586018155 >> 172.21.15.82:0/586018155 conn(0x7f1c580fd9c0 msgr2=0x7f1c580ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:15:32.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.307+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/586018155 shutdown_connections 2024-05-03T17:15:32.310 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:15:32.307+0000 7f1c5ffc1700 1 -- 172.21.15.82:0/586018155 wait complete. 2024-05-03T17:15:32.892 DEBUG:teuthology.orchestra.run.smithi082:osd.2> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.2.service 2024-05-03T17:15:32.895 INFO:tasks.cephadm:Deploying osd.3 on smithi067 with /dev/vg_nvme/lv_3... 2024-05-03T17:15:32.895 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-03T17:15:33.140 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:15:33.450 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:33 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:33.450 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:33 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:33.450 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:33 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:33.450 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:33 smithi082 ceph-mon[33535]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:33.450 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:33 smithi082 ceph-mon[33535]: from='osd.2 [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T17:15:33.450 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:33 smithi082 ceph-mon[33535]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T17:15:33.514 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:33 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:33.514 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:33 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:33.514 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:33 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:33.515 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:33 smithi067 ceph-mon[29731]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:33.515 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:33 smithi067 ceph-mon[29731]: from='osd.2 [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T17:15:33.515 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:33 smithi067 ceph-mon[29731]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T17:15:33.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:33 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:33.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:33 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:33.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:33 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:33.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:33 smithi070 ceph-mon[33250]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:33.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:33 smithi070 ceph-mon[33250]: from='osd.2 [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T17:15:33.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:33 smithi070 ceph-mon[33250]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-03T17:15:34.419 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:34 smithi082 ceph-mon[33535]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-03T17:15:34.419 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:34 smithi082 ceph-mon[33535]: osdmap e16: 3 total, 2 up, 3 in 2024-05-03T17:15:34.419 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:34.419 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:34 smithi082 ceph-mon[33535]: from='osd.2 [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:15:34.419 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:34 smithi082 ceph-mon[33535]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:15:34.419 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:15:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2[38836]: 2024-05-03T17:15:34.268+0000 7fed53f33700 -1 osd.2 0 waiting for initial osdmap 2024-05-03T17:15:34.419 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:15:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2[38836]: 2024-05-03T17:15:34.273+0000 7fed4fd26700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T17:15:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:34 smithi070 ceph-mon[33250]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-03T17:15:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:34 smithi070 ceph-mon[33250]: osdmap e16: 3 total, 2 up, 3 in 2024-05-03T17:15:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:34 smithi070 ceph-mon[33250]: from='osd.2 [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:15:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:34 smithi070 ceph-mon[33250]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:15:34.599 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:34 smithi067 ceph-mon[29731]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-03T17:15:34.600 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:34 smithi067 ceph-mon[29731]: osdmap e16: 3 total, 2 up, 3 in 2024-05-03T17:15:34.600 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:34.600 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:34 smithi067 ceph-mon[29731]: from='osd.2 [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:15:34.600 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:34 smithi067 ceph-mon[29731]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:15:35.204 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-03T17:15:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-03T17:15:35.205 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-05-03T17:15:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-05-03T17:15:35.205 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01125 s, 932 MB/s 2024-05-03T17:15:35.205 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-05-03T17:15:35.496 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:35 smithi082 ceph-mon[33535]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T17:15:35.496 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:35 smithi082 ceph-mon[33535]: osdmap e17: 3 total, 2 up, 3 in 2024-05-03T17:15:35.496 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:35 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:35.496 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:35 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:35.496 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:35 smithi082 ceph-mon[33535]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:35.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:35 smithi067 ceph-mon[29731]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T17:15:35.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:35 smithi067 ceph-mon[29731]: osdmap e17: 3 total, 2 up, 3 in 2024-05-03T17:15:35.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:35 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:35.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:35 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:35.516 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:35 smithi067 ceph-mon[29731]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:35.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:35 smithi070 ceph-mon[33250]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T17:15:35.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:35 smithi070 ceph-mon[33250]: osdmap e17: 3 total, 2 up, 3 in 2024-05-03T17:15:35.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:35 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:35.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:35 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:35.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:35 smithi070 ceph-mon[33250]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-03T17:15:35.920 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_3 2024-05-03T17:15:36.172 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:15:36.525 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: purged_snaps scrub starts 2024-05-03T17:15:36.526 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: purged_snaps scrub ok 2024-05-03T17:15:36.526 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:36.526 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: osd.2 [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264] boot 2024-05-03T17:15:36.526 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: osdmap e18: 3 total, 3 up, 3 in 2024-05-03T17:15:36.526 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:36.526 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: Detected new or changed devices on smithi082 2024-05-03T17:15:36.526 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:36.526 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:36.526 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:15:36.526 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi082 to 17088M 2024-05-03T17:15:36.526 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:36.527 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:36.527 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:36.527 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:36.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: purged_snaps scrub starts 2024-05-03T17:15:36.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: purged_snaps scrub ok 2024-05-03T17:15:36.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:36.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: osd.2 [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264] boot 2024-05-03T17:15:36.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: osdmap e18: 3 total, 3 up, 3 in 2024-05-03T17:15:36.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:36.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: Detected new or changed devices on smithi082 2024-05-03T17:15:36.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:36.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:36.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:15:36.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi082 to 17088M 2024-05-03T17:15:36.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:36.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:36.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:36.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:36.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: purged_snaps scrub starts 2024-05-03T17:15:36.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: purged_snaps scrub ok 2024-05-03T17:15:36.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:36.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: osd.2 [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264] boot 2024-05-03T17:15:36.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: osdmap e18: 3 total, 3 up, 3 in 2024-05-03T17:15:36.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-03T17:15:36.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: Detected new or changed devices on smithi082 2024-05-03T17:15:36.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:36.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:36.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:15:36.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi082 to 17088M 2024-05-03T17:15:36.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:36.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:36.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:36.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:37.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:37 smithi070 ceph-mon[33250]: osdmap e19: 3 total, 3 up, 3 in 2024-05-03T17:15:37.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:37 smithi070 ceph-mon[33250]: pgmap v84: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:37.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:37 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-03T17:15:37.647 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:37 smithi067 ceph-mon[29731]: osdmap e19: 3 total, 3 up, 3 in 2024-05-03T17:15:37.647 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:37 smithi067 ceph-mon[29731]: pgmap v84: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:37.647 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:37 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-03T17:15:37.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:37 smithi082 ceph-mon[33535]: osdmap e19: 3 total, 3 up, 3 in 2024-05-03T17:15:37.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:37 smithi082 ceph-mon[33535]: pgmap v84: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:37.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:37 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-03T17:15:37.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.866+0000 7fa659454700 1 -- 172.21.15.67:0/947007560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa654103fa0 msgr2=0x7fa65410a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:15:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.866+0000 7fa659454700 1 --2- 172.21.15.67:0/947007560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa654103fa0 0x7fa65410a850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fa6440099c0 tx=0x7fa644009cd0 comp rx=0 tx=0).stop 2024-05-03T17:15:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.866+0000 7fa659454700 1 -- 172.21.15.67:0/947007560 shutdown_connections 2024-05-03T17:15:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.866+0000 7fa659454700 1 --2- 172.21.15.67:0/947007560 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa654103fa0 0x7fa65410a850 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.866+0000 7fa659454700 1 --2- 172.21.15.67:0/947007560 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa654102400 0x7fa654102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.866+0000 7fa659454700 1 --2- 172.21.15.67:0/947007560 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa654103600 0x7fa654103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.866+0000 7fa659454700 1 -- 172.21.15.67:0/947007560 >> 172.21.15.67:0/947007560 conn(0x7fa6540fd9a0 msgr2=0x7fa6540ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:15:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.866+0000 7fa659454700 1 -- 172.21.15.67:0/947007560 shutdown_connections 2024-05-03T17:15:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.866+0000 7fa659454700 1 -- 172.21.15.67:0/947007560 wait complete. 2024-05-03T17:15:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa659454700 1 Processor -- start 2024-05-03T17:15:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa659454700 1 -- start start 2024-05-03T17:15:37.869 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa659454700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa654102400 0x7fa6541995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa659454700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa654103600 0x7fa654199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa659454700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa654103fa0 0x7fa6541a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa659454700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa65410de90 con 0x7fa654103fa0 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa659454700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa65410dd10 con 0x7fa654103600 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa659454700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa65410e010 con 0x7fa654102400 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa6537fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa654103fa0 0x7fa6541a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa6537fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa654103fa0 0x7fa6541a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41220/0 (socket says 172.21.15.67:41220) 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa6537fe700 1 -- 172.21.15.67:0/2112526087 learned_addr learned my addr 172.21.15.67:0/2112526087 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa652ffd700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa654102400 0x7fa6541995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa6527fc700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa654103600 0x7fa654199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa6537fe700 1 -- 172.21.15.67:0/2112526087 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa654102400 msgr2=0x7fa6541995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa6537fe700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa654102400 0x7fa6541995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa6537fe700 1 -- 172.21.15.67:0/2112526087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa654103600 msgr2=0x7fa654199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:15:37.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa6537fe700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa654103600 0x7fa654199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.867+0000 7fa6537fe700 1 -- 172.21.15.67:0/2112526087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6541a1220 con 0x7fa654103fa0 2024-05-03T17:15:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.868+0000 7fa6537fe700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa654103fa0 0x7fa6541a0b80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa644005910 tx=0x7fa6440047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:15:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.868+0000 7fa64bfff700 1 -- 172.21.15.67:0/2112526087 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa644027070 con 0x7fa654103fa0 2024-05-03T17:15:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.868+0000 7fa659454700 1 -- 172.21.15.67:0/2112526087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6541a14b0 con 0x7fa654103fa0 2024-05-03T17:15:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.868+0000 7fa64bfff700 1 -- 172.21.15.67:0/2112526087 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa644004020 con 0x7fa654103fa0 2024-05-03T17:15:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.868+0000 7fa659454700 1 -- 172.21.15.67:0/2112526087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6541a1960 con 0x7fa654103fa0 2024-05-03T17:15:37.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.868+0000 7fa64bfff700 1 -- 172.21.15.67:0/2112526087 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa644022d20 con 0x7fa654103fa0 2024-05-03T17:15:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.869+0000 7fa659454700 1 -- 172.21.15.67:0/2112526087 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa65406a190 con 0x7fa654103fa0 2024-05-03T17:15:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.870+0000 7fa64bfff700 1 -- 172.21.15.67:0/2112526087 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 15) ==== 149596+0+0 (secure 0 0 0) 0x7fa64402a070 con 0x7fa654103fa0 2024-05-03T17:15:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.871+0000 7fa64bfff700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa6400b27f0 0x7fa6400b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:15:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.871+0000 7fa652ffd700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa6400b27f0 0x7fa6400b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:15:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.871+0000 7fa64bfff700 1 -- 172.21.15.67:0/2112526087 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3226+0+0 (secure 0 0 0) 0x7fa6440e2060 con 0x7fa654103fa0 2024-05-03T17:15:37.872 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.872+0000 7fa652ffd700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa6400b27f0 0x7fa6400b4cb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa654103460 tx=0x7fa64c007a30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:15:37.875 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:37.875+0000 7fa64bfff700 1 -- 172.21.15.67:0/2112526087 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa644031070 con 0x7fa654103fa0 2024-05-03T17:15:38.101 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:38.100+0000 7fa659454700 1 -- 172.21.15.67:0/2112526087 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa6540ffb30 con 0x7fa6400b27f0 2024-05-03T17:15:38.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-03T17:15:38.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 ceph-mon[29731]: osdmap e20: 3 total, 3 up, 3 in 2024-05-03T17:15:38.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T17:15:38.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:15:38.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:15:38.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:38.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-03T17:15:38.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 ceph-mon[33250]: osdmap e20: 3 total, 3 up, 3 in 2024-05-03T17:15:38.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T17:15:38.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:15:38.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:15:38.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:38.703 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-03T17:15:38.703 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 ceph-mon[33535]: osdmap e20: 3 total, 3 up, 3 in 2024-05-03T17:15:38.703 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T17:15:38.703 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:15:38.703 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:15:38.703 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:38.913 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49468]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T17:15:38.913 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49468]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T17:15:38.913 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49468]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T17:15:38.913 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49468]: pam_unix(sudo:session): session closed for user root 2024-05-03T17:15:38.913 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49471]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T17:15:38.913 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49471]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T17:15:38.913 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49471]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T17:15:38.913 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49471]: pam_unix(sudo:session): session closed for user root 2024-05-03T17:15:38.913 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49421]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T17:15:38.914 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49421]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T17:15:38.914 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49421]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T17:15:38.914 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49421]: pam_unix(sudo:session): session closed for user root 2024-05-03T17:15:38.914 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49438]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T17:15:38.914 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49438]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T17:15:38.914 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49438]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T17:15:38.914 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:15:38 smithi067 sudo[49438]: pam_unix(sudo:session): session closed for user root 2024-05-03T17:15:39.015 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42750]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T17:15:39.015 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42750]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T17:15:39.015 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42750]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T17:15:39.015 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42750]: pam_unix(sudo:session): session closed for user root 2024-05-03T17:15:39.015 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42753]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T17:15:39.015 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42753]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T17:15:39.015 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42753]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T17:15:39.015 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42753]: pam_unix(sudo:session): session closed for user root 2024-05-03T17:15:39.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42756]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T17:15:39.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42756]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T17:15:39.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42756]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T17:15:39.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42756]: pam_unix(sudo:session): session closed for user root 2024-05-03T17:15:39.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42759]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T17:15:39.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42759]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T17:15:39.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42759]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T17:15:39.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:38 smithi082 sudo[42759]: pam_unix(sudo:session): session closed for user root 2024-05-03T17:15:39.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41996]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T17:15:39.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41996]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T17:15:39.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41996]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T17:15:39.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41996]: pam_unix(sudo:session): session closed for user root 2024-05-03T17:15:39.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41999]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T17:15:39.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41999]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T17:15:39.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41999]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T17:15:39.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41999]: pam_unix(sudo:session): session closed for user root 2024-05-03T17:15:39.059 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41990]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-03T17:15:39.059 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41990]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T17:15:39.059 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41990]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T17:15:39.059 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41990]: pam_unix(sudo:session): session closed for user root 2024-05-03T17:15:39.059 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41993]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-03T17:15:39.059 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41993]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-03T17:15:39.059 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41993]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-03T17:15:39.059 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:15:38 smithi070 sudo[41993]: pam_unix(sudo:session): session closed for user root 2024-05-03T17:15:39.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:15:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-03T17:15:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: osdmap e21: 3 total, 3 up, 3 in 2024-05-03T17:15:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T17:15:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T17:15:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:15:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:15:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:15:39.560 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T17:15:39.560 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:15:39.560 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:15:39.560 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:15:39.560 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T17:15:39.560 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T17:15:39.560 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:15:39.560 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:15:39.560 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:15:39.560 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:39 smithi070 ceph-mon[33250]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T17:15:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:15:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-03T17:15:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: osdmap e21: 3 total, 3 up, 3 in 2024-05-03T17:15:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:39.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T17:15:39.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T17:15:39.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:15:39.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:15:39.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:15:39.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T17:15:39.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:15:39.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:15:39.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:15:39.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T17:15:39.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T17:15:39.705 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:15:39.706 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:15:39.706 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:15:39.706 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:39 smithi067 ceph-mon[29731]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T17:15:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='client.14283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:15:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-03T17:15:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: osdmap e21: 3 total, 3 up, 3 in 2024-05-03T17:15:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: pgmap v87: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-03T17:15:39.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:39 smithi082 ceph-mon[33535]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-03T17:15:40.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:40 smithi067 ceph-mon[29731]: osdmap e22: 3 total, 3 up, 3 in 2024-05-03T17:15:40.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:40 smithi070 ceph-mon[33250]: osdmap e22: 3 total, 3 up, 3 in 2024-05-03T17:15:40.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:40 smithi082 ceph-mon[33535]: osdmap e22: 3 total, 3 up, 3 in 2024-05-03T17:15:41.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:41.305+0000 7fa64bfff700 1 -- 172.21.15.67:0/2112526087 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa6440a6e60 con 0x7fa654103fa0 2024-05-03T17:15:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:41 smithi070 ceph-mon[33250]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:41 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/4155105526' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14bfe6a6-0bc8-4766-8d36-fbaecc61a396"}]: dispatch 2024-05-03T17:15:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:41 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/4155105526' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14bfe6a6-0bc8-4766-8d36-fbaecc61a396"}]': finished 2024-05-03T17:15:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:41 smithi070 ceph-mon[33250]: osdmap e23: 4 total, 3 up, 4 in 2024-05-03T17:15:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:41 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:15:41.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:41 smithi067 ceph-mon[29731]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:41.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:41 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/4155105526' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14bfe6a6-0bc8-4766-8d36-fbaecc61a396"}]: dispatch 2024-05-03T17:15:41.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:41 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/4155105526' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14bfe6a6-0bc8-4766-8d36-fbaecc61a396"}]': finished 2024-05-03T17:15:41.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:41 smithi067 ceph-mon[29731]: osdmap e23: 4 total, 3 up, 4 in 2024-05-03T17:15:41.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:41 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:15:41.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:41 smithi082 ceph-mon[33535]: pgmap v89: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:41.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:41 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/4155105526' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "14bfe6a6-0bc8-4766-8d36-fbaecc61a396"}]: dispatch 2024-05-03T17:15:41.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:41 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/4155105526' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "14bfe6a6-0bc8-4766-8d36-fbaecc61a396"}]': finished 2024-05-03T17:15:41.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:41 smithi082 ceph-mon[33535]: osdmap e23: 4 total, 3 up, 4 in 2024-05-03T17:15:41.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:41 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:15:42.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:42 smithi070 ceph-mon[33250]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-05-03T17:15:42.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:42 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3166404794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:15:42.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:42 smithi067 ceph-mon[29731]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-05-03T17:15:42.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:42 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3166404794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:15:42.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:42 smithi082 ceph-mon[33535]: mgrmap e16: x(active, since 2m), standbys: y, z 2024-05-03T17:15:42.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:42 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3166404794' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:15:43.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:43 smithi070 ceph-mon[33250]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:43.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:43 smithi067 ceph-mon[29731]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:43.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:43 smithi082 ceph-mon[33535]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:45.636 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:45 smithi067 ceph-mon[29731]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:45.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:45 smithi082 ceph-mon[33535]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:45.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:45 smithi070 ceph-mon[33250]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:47.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:47 smithi067 ceph-mon[29731]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:47.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:47 smithi082 ceph-mon[33535]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:47.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:47 smithi070 ceph-mon[33250]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:49.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:49 smithi067 ceph-mon[29731]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:49.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-03T17:15:49.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:49.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:49 smithi067 ceph-mon[29731]: Deploying daemon osd.3 on smithi067 2024-05-03T17:15:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:49 smithi082 ceph-mon[33535]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-03T17:15:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:49.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:49 smithi082 ceph-mon[33535]: Deploying daemon osd.3 on smithi067 2024-05-03T17:15:49.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:49 smithi070 ceph-mon[33250]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:49.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-03T17:15:49.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:49.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:49 smithi070 ceph-mon[33250]: Deploying daemon osd.3 on smithi067 2024-05-03T17:15:51.606 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:51 smithi067 ceph-mon[29731]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:51.678 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:51 smithi082 ceph-mon[33535]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:51.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:51 smithi070 ceph-mon[33250]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:53.456 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:53 smithi067 ceph-mon[29731]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:53.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:53 smithi082 ceph-mon[33535]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:53.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:53 smithi070 ceph-mon[33250]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:56.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:55 smithi067 ceph-mon[29731]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:56.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:56.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:56.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:56.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:55 smithi082 ceph-mon[33535]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:56.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:56.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:56.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:56.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:55 smithi070 ceph-mon[33250]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:56.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:56.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:56.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:58.172 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:57 smithi067 ceph-mon[29731]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:58.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:57 smithi082 ceph-mon[33535]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:58.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:57 smithi070 ceph-mon[33250]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:58.925 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.924+0000 7fa64bfff700 1 -- 172.21.15.67:0/2112526087 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa6540ffb30 con 0x7fa6400b27f0 2024-05-03T17:15:58.926 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 3 on host 'smithi067' 2024-05-03T17:15:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.927+0000 7fa659454700 1 -- 172.21.15.67:0/2112526087 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa6400b27f0 msgr2=0x7fa6400b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:15:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.927+0000 7fa659454700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa6400b27f0 0x7fa6400b4cb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa654103460 tx=0x7fa64c007a30 comp rx=0 tx=0).stop 2024-05-03T17:15:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.927+0000 7fa659454700 1 -- 172.21.15.67:0/2112526087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa654103fa0 msgr2=0x7fa6541a0b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:15:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.927+0000 7fa659454700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa654103fa0 0x7fa6541a0b80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa644005910 tx=0x7fa6440047b0 comp rx=0 tx=0).stop 2024-05-03T17:15:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.927+0000 7fa659454700 1 -- 172.21.15.67:0/2112526087 shutdown_connections 2024-05-03T17:15:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.927+0000 7fa659454700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa6400b27f0 0x7fa6400b4cb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.927+0000 7fa659454700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa654103fa0 0x7fa6541a0b80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.927+0000 7fa659454700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa654102400 0x7fa6541995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.927+0000 7fa659454700 1 --2- 172.21.15.67:0/2112526087 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa654103600 0x7fa654199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:15:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.927+0000 7fa659454700 1 -- 172.21.15.67:0/2112526087 >> 172.21.15.67:0/2112526087 conn(0x7fa6540fd9a0 msgr2=0x7fa6540ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:15:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.927+0000 7fa659454700 1 -- 172.21.15.67:0/2112526087 shutdown_connections 2024-05-03T17:15:58.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:15:58.927+0000 7fa659454700 1 -- 172.21.15.67:0/2112526087 wait complete. 2024-05-03T17:15:59.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:59.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:59.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:59 smithi067 ceph-mon[29731]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:59.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:59 smithi067 ceph-mon[29731]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-03T17:15:59.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:59.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:15:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:59.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:59.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:59 smithi082 ceph-mon[33535]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:59.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:59 smithi082 ceph-mon[33535]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-03T17:15:59.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:59.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:15:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:15:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:15:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:59 smithi070 ceph-mon[33250]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:15:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:59 smithi070 ceph-mon[33250]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-03T17:15:59.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:15:59.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:15:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:15:59.623 DEBUG:teuthology.orchestra.run.smithi067:osd.3> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.3.service 2024-05-03T17:15:59.626 INFO:tasks.cephadm:Deploying osd.4 on smithi070 with /dev/vg_nvme/lv_3... 2024-05-03T17:15:59.626 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-03T17:15:59.872 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T17:16:00.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:00 smithi070 ceph-mon[33250]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-03T17:16:00.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:00 smithi070 ceph-mon[33250]: osdmap e24: 4 total, 3 up, 4 in 2024-05-03T17:16:00.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:00 smithi070 ceph-mon[33250]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:16:00.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:00 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:16:00.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:00 smithi067 ceph-mon[29731]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-03T17:16:00.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:00 smithi067 ceph-mon[29731]: osdmap e24: 4 total, 3 up, 4 in 2024-05-03T17:16:00.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:00 smithi067 ceph-mon[29731]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:16:00.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:00 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:16:00.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:00 smithi082 ceph-mon[33535]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-03T17:16:00.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:00 smithi082 ceph-mon[33535]: osdmap e24: 4 total, 3 up, 4 in 2024-05-03T17:16:00.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:00 smithi082 ceph-mon[33535]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:16:00.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:00 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:16:00.900 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:16:00 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3[51618]: 2024-05-03T17:16:00.611+0000 7fe779146700 -1 osd.3 0 waiting for initial osdmap 2024-05-03T17:16:00.900 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:16:00 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3[51618]: 2024-05-03T17:16:00.618+0000 7fe770730700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T17:16:01.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:01 smithi067 ceph-mon[29731]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T17:16:01.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:01 smithi067 ceph-mon[29731]: osdmap e25: 4 total, 3 up, 4 in 2024-05-03T17:16:01.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:01 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:16:01.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:01 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:16:01.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:01 smithi067 ceph-mon[29731]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:16:01.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:01 smithi067 ceph-mon[29731]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' 2024-05-03T17:16:01.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:01 smithi067 ceph-mon[29731]: osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723] boot 2024-05-03T17:16:01.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:01 smithi067 ceph-mon[29731]: osdmap e26: 4 total, 4 up, 4 in 2024-05-03T17:16:01.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:01 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:16:01.267 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:01 smithi070 ceph-mon[33250]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T17:16:01.268 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:01 smithi070 ceph-mon[33250]: osdmap e25: 4 total, 3 up, 4 in 2024-05-03T17:16:01.268 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:01 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:16:01.268 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:01 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:16:01.268 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:01 smithi070 ceph-mon[33250]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:16:01.268 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:01 smithi070 ceph-mon[33250]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' 2024-05-03T17:16:01.268 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:01 smithi070 ceph-mon[33250]: osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723] boot 2024-05-03T17:16:01.268 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:01 smithi070 ceph-mon[33250]: osdmap e26: 4 total, 4 up, 4 in 2024-05-03T17:16:01.268 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:01 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:16:01.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:01 smithi082 ceph-mon[33535]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T17:16:01.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:01 smithi082 ceph-mon[33535]: osdmap e25: 4 total, 3 up, 4 in 2024-05-03T17:16:01.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:01 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:16:01.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:01 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:16:01.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:01 smithi082 ceph-mon[33535]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-03T17:16:01.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:01 smithi082 ceph-mon[33535]: from='osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723]' entity='osd.3' 2024-05-03T17:16:01.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:01 smithi082 ceph-mon[33535]: osd.3 [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723] boot 2024-05-03T17:16:01.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:01 smithi082 ceph-mon[33535]: osdmap e26: 4 total, 4 up, 4 in 2024-05-03T17:16:01.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:01 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-03T17:16:01.948 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-03T17:16:01.948 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-03T17:16:01.948 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-05-03T17:16:01.949 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-05-03T17:16:01.949 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00866134 s, 1.2 GB/s 2024-05-03T17:16:01.949 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-05-03T17:16:02.272 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:02 smithi070 ceph-mon[33250]: purged_snaps scrub starts 2024-05-03T17:16:02.272 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:02 smithi070 ceph-mon[33250]: purged_snaps scrub ok 2024-05-03T17:16:02.272 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:02 smithi070 ceph-mon[33250]: osdmap e27: 4 total, 4 up, 4 in 2024-05-03T17:16:02.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:02 smithi067 ceph-mon[29731]: purged_snaps scrub starts 2024-05-03T17:16:02.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:02 smithi067 ceph-mon[29731]: purged_snaps scrub ok 2024-05-03T17:16:02.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:02 smithi067 ceph-mon[29731]: osdmap e27: 4 total, 4 up, 4 in 2024-05-03T17:16:02.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:02 smithi082 ceph-mon[33535]: purged_snaps scrub starts 2024-05-03T17:16:02.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:02 smithi082 ceph-mon[33535]: purged_snaps scrub ok 2024-05-03T17:16:02.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:02 smithi082 ceph-mon[33535]: osdmap e27: 4 total, 4 up, 4 in 2024-05-03T17:16:02.608 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_3 2024-05-03T17:16:02.849 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T17:16:04.114 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:03 smithi070 ceph-mon[33250]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:04.114 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:03 smithi070 ceph-mon[33250]: Detected new or changed devices on smithi067 2024-05-03T17:16:04.114 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:03 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:04.114 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:03 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:04.114 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:03 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:04.114 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:03 smithi070 ceph-mon[33250]: osdmap e28: 4 total, 4 up, 4 in 2024-05-03T17:16:04.114 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:03 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:04.114 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:03 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi067 to 8544M 2024-05-03T17:16:04.115 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:03 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:04.115 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:03 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:04.115 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:03 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:04.115 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:03 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:04.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:03 smithi067 ceph-mon[29731]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:04.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:03 smithi067 ceph-mon[29731]: Detected new or changed devices on smithi067 2024-05-03T17:16:04.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:03 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:04.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:03 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:04.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:03 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:04.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:03 smithi067 ceph-mon[29731]: osdmap e28: 4 total, 4 up, 4 in 2024-05-03T17:16:04.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:03 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:04.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:03 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi067 to 8544M 2024-05-03T17:16:04.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:03 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:04.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:03 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:04.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:03 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:04.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:03 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:04.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:03 smithi082 ceph-mon[33535]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:04.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:03 smithi082 ceph-mon[33535]: Detected new or changed devices on smithi067 2024-05-03T17:16:04.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:03 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:04.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:03 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:04.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:03 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:04.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:03 smithi082 ceph-mon[33535]: osdmap e28: 4 total, 4 up, 4 in 2024-05-03T17:16:04.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:03 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:04.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:03 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi067 to 8544M 2024-05-03T17:16:04.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:03 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:04.268 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:03 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:04.268 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:03 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:04.268 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:03 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.351+0000 7faa4a192700 1 -- 172.21.15.70:0/494454968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa441068c0 msgr2=0x7faa44108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.351+0000 7faa4a192700 1 --2- 172.21.15.70:0/494454968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa441068c0 0x7faa44108cb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7faa38009990 tx=0x7faa38009ca0 comp rx=0 tx=0).stop 2024-05-03T17:16:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.351+0000 7faa4a192700 1 -- 172.21.15.70:0/494454968 shutdown_connections 2024-05-03T17:16:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.351+0000 7faa4a192700 1 --2- 172.21.15.70:0/494454968 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa441091f0 0x7faa4410b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.351+0000 7faa4a192700 1 --2- 172.21.15.70:0/494454968 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa44103f90 0x7faa44106380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.351+0000 7faa4a192700 1 --2- 172.21.15.70:0/494454968 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa441068c0 0x7faa44108cb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.351+0000 7faa4a192700 1 -- 172.21.15.70:0/494454968 >> 172.21.15.70:0/494454968 conn(0x7faa440fd9a0 msgr2=0x7faa440ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:16:04.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.351+0000 7faa4a192700 1 -- 172.21.15.70:0/494454968 shutdown_connections 2024-05-03T17:16:04.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.351+0000 7faa4a192700 1 -- 172.21.15.70:0/494454968 wait complete. 2024-05-03T17:16:04.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.351+0000 7faa4a192700 1 Processor -- start 2024-05-03T17:16:04.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa4a192700 1 -- start start 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa4a192700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa44103f90 0x7faa441038f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa4a192700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa441068c0 0x7faa44101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa4a192700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa441091f0 0x7faa44102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa4a192700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7faa4410f010 con 0x7faa441091f0 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa4a192700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7faa4410ee90 con 0x7faa44103f90 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa4a192700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7faa4410f190 con 0x7faa441068c0 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa437fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa44103f90 0x7faa441038f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa437fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa44103f90 0x7faa441038f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:47820/0 (socket says 172.21.15.70:47820) 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa437fe700 1 -- 172.21.15.70:0/3449131355 learned_addr learned my addr 172.21.15.70:0/3449131355 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa42ffd700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa441068c0 0x7faa44101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa437fe700 1 -- 172.21.15.70:0/3449131355 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa441068c0 msgr2=0x7faa44101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa437fe700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa441068c0 0x7faa44101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:04.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa437fe700 1 -- 172.21.15.70:0/3449131355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa441091f0 msgr2=0x7faa44102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:04.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa43fff700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa441091f0 0x7faa44102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:16:04.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa437fe700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa441091f0 0x7faa44102480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:04.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa437fe700 1 -- 172.21.15.70:0/3449131355 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa44102d30 con 0x7faa44103f90 2024-05-03T17:16:04.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.352+0000 7faa437fe700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa44103f90 0x7faa441038f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faa3400d980 tx=0x7faa3400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:16:04.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.353+0000 7faa40ff9700 1 -- 172.21.15.70:0/3449131355 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34017070 con 0x7faa44103f90 2024-05-03T17:16:04.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.353+0000 7faa40ff9700 1 -- 172.21.15.70:0/3449131355 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa34005020 con 0x7faa44103f90 2024-05-03T17:16:04.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.353+0000 7faa4a192700 1 -- 172.21.15.70:0/3449131355 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa441a37c0 con 0x7faa44103f90 2024-05-03T17:16:04.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.353+0000 7faa40ff9700 1 -- 172.21.15.70:0/3449131355 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa34009b50 con 0x7faa44103f90 2024-05-03T17:16:04.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.353+0000 7faa4a192700 1 -- 172.21.15.70:0/3449131355 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa441a3c80 con 0x7faa44103f90 2024-05-03T17:16:04.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.354+0000 7faa4a192700 1 -- 172.21.15.70:0/3449131355 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4406a190 con 0x7faa44103f90 2024-05-03T17:16:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.356+0000 7faa40ff9700 1 -- 172.21.15.70:0/3449131355 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7faa340051d0 con 0x7faa44103f90 2024-05-03T17:16:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.356+0000 7faa40ff9700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7faa2c0b2880 0x7faa2c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:16:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.356+0000 7faa40ff9700 1 -- 172.21.15.70:0/3449131355 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7faa340e1380 con 0x7faa44103f90 2024-05-03T17:16:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.356+0000 7faa42ffd700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7faa2c0b2880 0x7faa2c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:16:04.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.357+0000 7faa42ffd700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7faa2c0b2880 0x7faa2c0b4d40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faa3800b5c0 tx=0x7faa38005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:16:04.361 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.360+0000 7faa40ff9700 1 -- 172.21.15.70:0/3449131355 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7faa34020020 con 0x7faa44103f90 2024-05-03T17:16:04.575 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:04.575+0000 7faa4a192700 1 -- 172.21.15.70:0/3449131355 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7faa440638e0 con 0x7faa2c0b2880 2024-05-03T17:16:05.175 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:16:05.175 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:16:05.175 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:05.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:16:05.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:16:05.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:05.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:16:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:16:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:06.177 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:05 smithi070 ceph-mon[33250]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:06.177 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:05 smithi070 ceph-mon[33250]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:16:06.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:05 smithi067 ceph-mon[29731]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:06.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:05 smithi067 ceph-mon[29731]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:16:06.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:05 smithi082 ceph-mon[33535]: pgmap v107: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:06.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:05 smithi082 ceph-mon[33535]: from='client.24194 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:16:08.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:07 smithi067 ceph-mon[29731]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:08.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:07 smithi067 ceph-mon[29731]: from='client.? 172.21.15.70:0/335647287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a44aeb8-aabf-4824-be65-2c7170dbee52"}]: dispatch 2024-05-03T17:16:08.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:07 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a44aeb8-aabf-4824-be65-2c7170dbee52"}]: dispatch 2024-05-03T17:16:08.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:07 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a44aeb8-aabf-4824-be65-2c7170dbee52"}]': finished 2024-05-03T17:16:08.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:07 smithi067 ceph-mon[29731]: osdmap e29: 5 total, 4 up, 5 in 2024-05-03T17:16:08.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:07 smithi082 ceph-mon[33535]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:07 smithi082 ceph-mon[33535]: from='client.? 172.21.15.70:0/335647287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a44aeb8-aabf-4824-be65-2c7170dbee52"}]: dispatch 2024-05-03T17:16:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:07 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a44aeb8-aabf-4824-be65-2c7170dbee52"}]: dispatch 2024-05-03T17:16:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:07 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a44aeb8-aabf-4824-be65-2c7170dbee52"}]': finished 2024-05-03T17:16:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:07 smithi082 ceph-mon[33535]: osdmap e29: 5 total, 4 up, 5 in 2024-05-03T17:16:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:07 smithi070 ceph-mon[33250]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:07 smithi070 ceph-mon[33250]: from='client.? 172.21.15.70:0/335647287' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a44aeb8-aabf-4824-be65-2c7170dbee52"}]: dispatch 2024-05-03T17:16:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:07 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a44aeb8-aabf-4824-be65-2c7170dbee52"}]: dispatch 2024-05-03T17:16:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:07 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a44aeb8-aabf-4824-be65-2c7170dbee52"}]': finished 2024-05-03T17:16:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:07 smithi070 ceph-mon[33250]: osdmap e29: 5 total, 4 up, 5 in 2024-05-03T17:16:08.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:09.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:08 smithi067 ceph-mon[29731]: from='client.? 172.21.15.70:0/3912474764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:16:09.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:08 smithi082 ceph-mon[33535]: from='client.? 172.21.15.70:0/3912474764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:16:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:08 smithi070 ceph-mon[33250]: from='client.? 172.21.15.70:0/3912474764' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:16:10.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:09 smithi067 ceph-mon[29731]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-03T17:16:10.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:09 smithi082 ceph-mon[33535]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-03T17:16:10.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:09 smithi070 ceph-mon[33250]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-03T17:16:12.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:11 smithi067 ceph-mon[29731]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-03T17:16:12.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:11 smithi082 ceph-mon[33535]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-03T17:16:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:11 smithi070 ceph-mon[33250]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-03T17:16:14.173 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:13 smithi070 ceph-mon[33250]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-03T17:16:14.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:13 smithi067 ceph-mon[29731]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-03T17:16:14.267 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:13 smithi082 ceph-mon[33535]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-03T17:16:15.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:14 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-03T17:16:15.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:14 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:15.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:14 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-03T17:16:15.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:14 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:15.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:14 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-03T17:16:15.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:14 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:16.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:15 smithi070 ceph-mon[33250]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:16:16.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:15 smithi070 ceph-mon[33250]: Deploying daemon osd.4 on smithi070 2024-05-03T17:16:16.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:15 smithi067 ceph-mon[29731]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:16:16.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:15 smithi067 ceph-mon[29731]: Deploying daemon osd.4 on smithi070 2024-05-03T17:16:16.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:15 smithi082 ceph-mon[33535]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:16:16.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:15 smithi082 ceph-mon[33535]: Deploying daemon osd.4 on smithi070 2024-05-03T17:16:18.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:17 smithi067 ceph-mon[29731]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:16:18.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:17 smithi082 ceph-mon[33535]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:16:18.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:17 smithi070 ceph-mon[33250]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:16:20.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:19 smithi067 ceph-mon[29731]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:20.207 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:19 smithi070 ceph-mon[33250]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:20.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:19 smithi082 ceph-mon[33535]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:21.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:20 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:16:21.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:20 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:21.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:20 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:21.236 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:20 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:16:21.236 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:20 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:21.237 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:20 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:21.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:20 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:16:21.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:20 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:21.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:20 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:22.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:21 smithi067 ceph-mon[29731]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:22.206 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:21 smithi070 ceph-mon[33250]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:22.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:21 smithi082 ceph-mon[33535]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:24.032 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:23 smithi070 ceph-mon[33250]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:24.032 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:24.032 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:24.032 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:24.032 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:24.032 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:24.092 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 4 on host 'smithi070' 2024-05-03T17:16:24.093 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.091+0000 7faa40ff9700 1 -- 172.21.15.70:0/3449131355 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7faa440638e0 con 0x7faa2c0b2880 2024-05-03T17:16:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.094+0000 7faa4a192700 1 -- 172.21.15.70:0/3449131355 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7faa2c0b2880 msgr2=0x7faa2c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.094+0000 7faa4a192700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7faa2c0b2880 0x7faa2c0b4d40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faa3800b5c0 tx=0x7faa38005fb0 comp rx=0 tx=0).stop 2024-05-03T17:16:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.094+0000 7faa4a192700 1 -- 172.21.15.70:0/3449131355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa44103f90 msgr2=0x7faa441038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.094+0000 7faa4a192700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa44103f90 0x7faa441038f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7faa3400d980 tx=0x7faa3400dc90 comp rx=0 tx=0).stop 2024-05-03T17:16:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.094+0000 7faa4a192700 1 -- 172.21.15.70:0/3449131355 shutdown_connections 2024-05-03T17:16:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.094+0000 7faa4a192700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7faa2c0b2880 0x7faa2c0b4d40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.094+0000 7faa4a192700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7faa441091f0 0x7faa44102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.094+0000 7faa4a192700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7faa44103f90 0x7faa441038f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.094+0000 7faa4a192700 1 --2- 172.21.15.70:0/3449131355 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7faa441068c0 0x7faa44101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.094+0000 7faa4a192700 1 -- 172.21.15.70:0/3449131355 >> 172.21.15.70:0/3449131355 conn(0x7faa440fd9a0 msgr2=0x7faa44104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:16:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.094+0000 7faa4a192700 1 -- 172.21.15.70:0/3449131355 shutdown_connections 2024-05-03T17:16:24.095 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:16:24.094+0000 7faa4a192700 1 -- 172.21.15.70:0/3449131355 wait complete. 2024-05-03T17:16:24.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:23 smithi082 ceph-mon[33535]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:24.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:24.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:24.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:24.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:24.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:24.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:23 smithi067 ceph-mon[29731]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:24.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:24.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:24.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:24.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:24.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:24.702 DEBUG:teuthology.orchestra.run.smithi070:osd.4> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.4.service 2024-05-03T17:16:24.705 INFO:tasks.cephadm:Deploying osd.5 on smithi082 with /dev/vg_nvme/lv_3... 2024-05-03T17:16:24.705 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-03T17:16:24.950 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:16:25.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:25.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:25.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:16:25.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:25 smithi070 ceph-mon[33250]: from='osd.4 [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T17:16:25.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:25 smithi070 ceph-mon[33250]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T17:16:25.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:25 smithi070 ceph-mon[33250]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:25.385 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:25.385 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:25.385 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:16:25.385 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:25 smithi082 ceph-mon[33535]: from='osd.4 [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T17:16:25.385 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:25 smithi082 ceph-mon[33535]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T17:16:25.385 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:25 smithi082 ceph-mon[33535]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:25.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:25.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:25.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:16:25.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:25 smithi067 ceph-mon[29731]: from='osd.4 [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T17:16:25.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:25 smithi067 ceph-mon[29731]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-03T17:16:25.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:25 smithi067 ceph-mon[29731]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:26.162 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:26 smithi082 ceph-mon[33535]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-03T17:16:26.163 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:26 smithi082 ceph-mon[33535]: from='osd.4 [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:16:26.163 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:26 smithi082 ceph-mon[33535]: osdmap e30: 5 total, 4 up, 5 in 2024-05-03T17:16:26.163 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:26 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:26.163 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:26 smithi082 ceph-mon[33535]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:16:26.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:26 smithi070 ceph-mon[33250]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-03T17:16:26.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:26 smithi070 ceph-mon[33250]: from='osd.4 [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:16:26.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:26 smithi070 ceph-mon[33250]: osdmap e30: 5 total, 4 up, 5 in 2024-05-03T17:16:26.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:26 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:26.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:26 smithi070 ceph-mon[33250]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:16:26.309 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:16:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4[44798]: 2024-05-03T17:16:26.079+0000 7fb0d9763700 -1 osd.4 0 waiting for initial osdmap 2024-05-03T17:16:26.309 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:16:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4[44798]: 2024-05-03T17:16:26.086+0000 7fb0d5556700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T17:16:26.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:26 smithi067 ceph-mon[29731]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-03T17:16:26.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:26 smithi067 ceph-mon[29731]: from='osd.4 [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:16:26.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:26 smithi067 ceph-mon[29731]: osdmap e30: 5 total, 4 up, 5 in 2024-05-03T17:16:26.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:26 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:26.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:26 smithi067 ceph-mon[29731]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:16:26.720 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-03T17:16:26.720 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-03T17:16:26.720 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-05-03T17:16:26.720 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-05-03T17:16:26.720 INFO:teuthology.orchestra.run.smithi082.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123876 s, 846 MB/s 2024-05-03T17:16:26.720 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-05-03T17:16:27.344 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:27 smithi082 ceph-mon[33535]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-05-03T17:16:27.344 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:27 smithi082 ceph-mon[33535]: osdmap e31: 5 total, 4 up, 5 in 2024-05-03T17:16:27.345 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:27 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:27.345 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:27 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:27.345 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:27 smithi082 ceph-mon[33535]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:27.346 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_3 2024-05-03T17:16:27.449 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:27 smithi070 ceph-mon[33250]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-05-03T17:16:27.449 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:27 smithi070 ceph-mon[33250]: osdmap e31: 5 total, 4 up, 5 in 2024-05-03T17:16:27.449 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:27 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:27.450 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:27 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:27.450 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:27 smithi070 ceph-mon[33250]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:27.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:27 smithi067 ceph-mon[29731]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-05-03T17:16:27.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:27 smithi067 ceph-mon[29731]: osdmap e31: 5 total, 4 up, 5 in 2024-05-03T17:16:27.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:27 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:27.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:27 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:27.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:27 smithi067 ceph-mon[29731]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 115 MiB used, 357 GiB / 358 GiB avail 2024-05-03T17:16:27.590 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: purged_snaps scrub starts 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: purged_snaps scrub ok 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: osd.4 [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828] boot 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: osdmap e32: 5 total, 5 up, 5 in 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: Detected new or changed devices on smithi070 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi070 to 8544M 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:28.309 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:28.310 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:28.310 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:28.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: purged_snaps scrub starts 2024-05-03T17:16:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: purged_snaps scrub ok 2024-05-03T17:16:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: osd.4 [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828] boot 2024-05-03T17:16:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: osdmap e32: 5 total, 5 up, 5 in 2024-05-03T17:16:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: Detected new or changed devices on smithi070 2024-05-03T17:16:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi070 to 8544M 2024-05-03T17:16:28.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:28.455 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:28.455 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:28.455 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:28.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: purged_snaps scrub starts 2024-05-03T17:16:28.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: purged_snaps scrub ok 2024-05-03T17:16:28.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:28.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: osd.4 [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828] boot 2024-05-03T17:16:28.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: osdmap e32: 5 total, 5 up, 5 in 2024-05-03T17:16:28.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-03T17:16:28.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: Detected new or changed devices on smithi070 2024-05-03T17:16:28.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:28.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:28.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:28.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:28.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi070 to 8544M 2024-05-03T17:16:28.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:28.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:28.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:28.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:29.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.148+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/382887362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbbc103500 msgr2=0x7fbbbc103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:29.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.148+0000 7fbbc1bd7700 1 --2- 172.21.15.82:0/382887362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbbc103500 0x7fbbbc103920 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbbac009990 tx=0x7fbbac009ca0 comp rx=0 tx=0).stop 2024-05-03T17:16:29.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.148+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/382887362 shutdown_connections 2024-05-03T17:16:29.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.148+0000 7fbbc1bd7700 1 --2- 172.21.15.82:0/382887362 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbbc1050a0 0x7fbbbc10b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:29.149 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.148+0000 7fbbc1bd7700 1 --2- 172.21.15.82:0/382887362 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbbc103500 0x7fbbbc103920 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:29.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.148+0000 7fbbc1bd7700 1 --2- 172.21.15.82:0/382887362 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbbc104700 0x7fbbbc104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:29.150 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.148+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/382887362 >> 172.21.15.82:0/382887362 conn(0x7fbbbc0feaa0 msgr2=0x7fbbbc100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:16:29.151 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.149+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/382887362 shutdown_connections 2024-05-03T17:16:29.151 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.149+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/382887362 wait complete. 2024-05-03T17:16:29.151 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbc1bd7700 1 Processor -- start 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbc1bd7700 1 -- start start 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbc1bd7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbbc103500 0x7fbbbc1996e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbc1bd7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbbc104700 0x7fbbbc199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbc1bd7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbbc1050a0 0x7fbbbc1a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbc1bd7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc10efd0 con 0x7fbbbc1050a0 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbc1bd7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc10ee50 con 0x7fbbbc103500 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbc1bd7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc10f150 con 0x7fbbbc104700 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbb3fff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbbc104700 0x7fbbbc199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbb3fff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbbc104700 0x7fbbbc199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:40838/0 (socket says 172.21.15.82:40838) 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbb3fff700 1 -- 172.21.15.82:0/36948917 learned_addr learned my addr 172.21.15.82:0/36948917 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbbb7fe700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbbc103500 0x7fbbbc1996e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbb3fff700 1 -- 172.21.15.82:0/36948917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbbc103500 msgr2=0x7fbbbc1996e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbb3fff700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbbc103500 0x7fbbbc1996e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbb3fff700 1 -- 172.21.15.82:0/36948917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbbc1050a0 msgr2=0x7fbbbc1a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:29.152 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbb3fff700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbbc1050a0 0x7fbbbc1a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:29.153 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbb3fff700 1 -- 172.21.15.82:0/36948917 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc1a1340 con 0x7fbbbc104700 2024-05-03T17:16:29.153 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.150+0000 7fbbb3fff700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbbc104700 0x7fbbbc199c20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbba400c950 tx=0x7fbba400cd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:16:29.153 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.151+0000 7fbbb97fa700 1 -- 172.21.15.82:0/36948917 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4007ac0 con 0x7fbbbc104700 2024-05-03T17:16:29.153 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.151+0000 7fbbb97fa700 1 -- 172.21.15.82:0/36948917 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbba4004ce0 con 0x7fbbbc104700 2024-05-03T17:16:29.153 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.151+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/36948917 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbbc1a1630 con 0x7fbbbc104700 2024-05-03T17:16:29.153 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.151+0000 7fbbb97fa700 1 -- 172.21.15.82:0/36948917 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4005550 con 0x7fbbbc104700 2024-05-03T17:16:29.153 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.151+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/36948917 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbbc1a1b70 con 0x7fbbbc104700 2024-05-03T17:16:29.153 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.152+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/36948917 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbbc06a190 con 0x7fbbbc104700 2024-05-03T17:16:29.153 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.153+0000 7fbbb97fa700 1 -- 172.21.15.82:0/36948917 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fbba4007c60 con 0x7fbbbc104700 2024-05-03T17:16:29.154 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.154+0000 7fbbb97fa700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fbba80b2880 0x7fbba80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:16:29.155 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.154+0000 7fbbbb7fe700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fbba80b2880 0x7fbba80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:16:29.155 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.154+0000 7fbbb97fa700 1 -- 172.21.15.82:0/36948917 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7fbba40e1260 con 0x7fbbbc104700 2024-05-03T17:16:29.155 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.154+0000 7fbbbb7fe700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fbba80b2880 0x7fbba80b4d40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0038a0 tx=0x7fbbac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:16:29.158 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.158+0000 7fbbb97fa700 1 -- 172.21.15.82:0/36948917 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fbba4014030 con 0x7fbbbc104700 2024-05-03T17:16:29.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:29 smithi082 ceph-mon[33535]: osdmap e33: 5 total, 5 up, 5 in 2024-05-03T17:16:29.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:29 smithi082 ceph-mon[33535]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:29.379 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:29.378+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/36948917 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fbbbc0638e0 con 0x7fbba80b2880 2024-05-03T17:16:29.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:29 smithi067 ceph-mon[29731]: osdmap e33: 5 total, 5 up, 5 in 2024-05-03T17:16:29.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:29 smithi067 ceph-mon[29731]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:29.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:29 smithi070 ceph-mon[33250]: osdmap e33: 5 total, 5 up, 5 in 2024-05-03T17:16:29.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:29 smithi070 ceph-mon[33250]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:30.240 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:30 smithi082 ceph-mon[33535]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:16:30.240 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:30 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:16:30.240 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:30 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:16:30.240 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:30 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:30.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:30 smithi067 ceph-mon[29731]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:16:30.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:30 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:16:30.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:30 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:16:30.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:30 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:30.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:30 smithi070 ceph-mon[33250]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:16:30.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:30 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:16:30.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:30 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:16:30.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:30 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:31.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:31 smithi082 ceph-mon[33535]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:31.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:31 smithi067 ceph-mon[29731]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:31.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:31 smithi070 ceph-mon[33250]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:32.413 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:32 smithi082 ceph-mon[33535]: from='client.? 172.21.15.82:0/4277899591' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6ac56df-466e-4096-8fbf-835f5b544297"}]: dispatch 2024-05-03T17:16:32.413 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:32 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6ac56df-466e-4096-8fbf-835f5b544297"}]: dispatch 2024-05-03T17:16:32.414 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:32 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6ac56df-466e-4096-8fbf-835f5b544297"}]': finished 2024-05-03T17:16:32.414 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:32 smithi082 ceph-mon[33535]: osdmap e34: 6 total, 5 up, 6 in 2024-05-03T17:16:32.414 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:32 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:32.429 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:32 smithi067 ceph-mon[29731]: from='client.? 172.21.15.82:0/4277899591' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6ac56df-466e-4096-8fbf-835f5b544297"}]: dispatch 2024-05-03T17:16:32.429 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:32 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6ac56df-466e-4096-8fbf-835f5b544297"}]: dispatch 2024-05-03T17:16:32.429 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:32 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6ac56df-466e-4096-8fbf-835f5b544297"}]': finished 2024-05-03T17:16:32.429 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:32 smithi067 ceph-mon[29731]: osdmap e34: 6 total, 5 up, 6 in 2024-05-03T17:16:32.429 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:32 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:32.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:32 smithi070 ceph-mon[33250]: from='client.? 172.21.15.82:0/4277899591' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6ac56df-466e-4096-8fbf-835f5b544297"}]: dispatch 2024-05-03T17:16:32.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:32 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a6ac56df-466e-4096-8fbf-835f5b544297"}]: dispatch 2024-05-03T17:16:32.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:32 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a6ac56df-466e-4096-8fbf-835f5b544297"}]': finished 2024-05-03T17:16:32.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:32 smithi070 ceph-mon[33250]: osdmap e34: 6 total, 5 up, 6 in 2024-05-03T17:16:32.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:32 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:33.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:33 smithi067 ceph-mon[29731]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:33.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:33 smithi067 ceph-mon[29731]: from='client.? 172.21.15.82:0/3181836709' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:16:33.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:33 smithi082 ceph-mon[33535]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:33.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:33 smithi082 ceph-mon[33535]: from='client.? 172.21.15.82:0/3181836709' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:16:33.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:33 smithi070 ceph-mon[33250]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:33.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:33 smithi070 ceph-mon[33250]: from='client.? 172.21.15.82:0/3181836709' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:16:35.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:35 smithi067 ceph-mon[29731]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:35.717 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:35 smithi082 ceph-mon[33535]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:35.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:35 smithi070 ceph-mon[33250]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:37.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:37 smithi082 ceph-mon[33535]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:37.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:37 smithi067 ceph-mon[29731]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:37.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:37 smithi070 ceph-mon[33250]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:39.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:39 smithi067 ceph-mon[29731]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:39.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-03T17:16:39.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:39.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:39 smithi082 ceph-mon[33535]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-03T17:16:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:39.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:39 smithi070 ceph-mon[33250]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:39.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-03T17:16:39.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:40.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:40 smithi067 ceph-mon[29731]: Deploying daemon osd.5 on smithi082 2024-05-03T17:16:40.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:40 smithi082 ceph-mon[33535]: Deploying daemon osd.5 on smithi082 2024-05-03T17:16:40.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:40 smithi070 ceph-mon[33250]: Deploying daemon osd.5 on smithi082 2024-05-03T17:16:41.498 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:41 smithi082 ceph-mon[33535]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:41.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:41 smithi067 ceph-mon[29731]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:41.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:41 smithi070 ceph-mon[33250]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:43.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:43 smithi067 ceph-mon[29731]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:43 smithi082 ceph-mon[33535]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:43 smithi070 ceph-mon[33250]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:46.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:45 smithi082 ceph-mon[33535]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:46.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:45 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:46.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:45 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:46.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:45 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:16:46.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:45 smithi070 ceph-mon[33250]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:46.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:45 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:46.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:45 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:46.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:45 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:16:46.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:45 smithi067 ceph-mon[29731]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:46.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:45 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:46.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:45 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:46.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:45 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:16:48.220 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:47 smithi082 ceph-mon[33535]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:48.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:47 smithi070 ceph-mon[33250]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:48.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:47 smithi067 ceph-mon[29731]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:48.950 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.948+0000 7fbbb97fa700 1 -- 172.21.15.82:0/36948917 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fbbbc0638e0 con 0x7fbba80b2880 2024-05-03T17:16:48.951 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 5 on host 'smithi082' 2024-05-03T17:16:48.952 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.951+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/36948917 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fbba80b2880 msgr2=0x7fbba80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:48.952 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.951+0000 7fbbc1bd7700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fbba80b2880 0x7fbba80b4d40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fbbac0038a0 tx=0x7fbbac005fb0 comp rx=0 tx=0).stop 2024-05-03T17:16:48.953 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.951+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/36948917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbbc104700 msgr2=0x7fbbbc199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:48.953 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.951+0000 7fbbc1bd7700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbbc104700 0x7fbbbc199c20 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbba400c950 tx=0x7fbba400cd10 comp rx=0 tx=0).stop 2024-05-03T17:16:48.953 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.951+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/36948917 shutdown_connections 2024-05-03T17:16:48.953 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.951+0000 7fbbc1bd7700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fbba80b2880 0x7fbba80b4d40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:48.953 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.951+0000 7fbbc1bd7700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbbbc1050a0 0x7fbbbc1a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:48.953 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.951+0000 7fbbc1bd7700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbbbc103500 0x7fbbbc1996e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:48.953 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.951+0000 7fbbc1bd7700 1 --2- 172.21.15.82:0/36948917 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbbbc104700 0x7fbbbc199c20 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:48.953 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.951+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/36948917 >> 172.21.15.82:0/36948917 conn(0x7fbbbc0feaa0 msgr2=0x7fbbbc107b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:16:48.953 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.951+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/36948917 shutdown_connections 2024-05-03T17:16:48.953 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:16:48.951+0000 7fbbc1bd7700 1 -- 172.21.15.82:0/36948917 wait complete. 2024-05-03T17:16:49.259 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.259 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.259 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:49.259 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:49.259 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.259 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:49 smithi082 ceph-mon[33535]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:49.259 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:49 smithi082 ceph-mon[33535]: from='osd.5 [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T17:16:49.260 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:49 smithi082 ceph-mon[33535]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T17:16:49.260 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.260 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.260 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:16:49.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:49.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:49.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:49 smithi070 ceph-mon[33250]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:49.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:49 smithi070 ceph-mon[33250]: from='osd.5 [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T17:16:49.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:49 smithi070 ceph-mon[33250]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T17:16:49.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:16:49.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:49.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:49.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:49 smithi067 ceph-mon[29731]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:49.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:49 smithi067 ceph-mon[29731]: from='osd.5 [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T17:16:49.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:49 smithi067 ceph-mon[29731]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-03T17:16:49.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:49.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:16:49.601 DEBUG:teuthology.orchestra.run.smithi082:osd.5> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.5.service 2024-05-03T17:16:49.603 INFO:tasks.cephadm:Deploying osd.6 on smithi067 with /dev/vg_nvme/lv_2... 2024-05-03T17:16:49.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-03T17:16:49.860 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:16:50.188 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:50 smithi082 ceph-mon[33535]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-03T17:16:50.188 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:50 smithi082 ceph-mon[33535]: osdmap e35: 6 total, 5 up, 6 in 2024-05-03T17:16:50.188 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:50.188 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:50 smithi082 ceph-mon[33535]: from='osd.5 [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:16:50.188 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:50 smithi082 ceph-mon[33535]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:16:50.189 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:16:50 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:16:50.083+0000 7fcfaf3f5700 -1 osd.5 0 waiting for initial osdmap 2024-05-03T17:16:50.189 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:16:50 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:16:50.090+0000 7fcfab1e8700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T17:16:50.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:50 smithi067 ceph-mon[29731]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-03T17:16:50.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:50 smithi067 ceph-mon[29731]: osdmap e35: 6 total, 5 up, 6 in 2024-05-03T17:16:50.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:50.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:50 smithi067 ceph-mon[29731]: from='osd.5 [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:16:50.261 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:50 smithi067 ceph-mon[29731]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:16:50.408 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:50 smithi070 ceph-mon[33250]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-03T17:16:50.408 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:50 smithi070 ceph-mon[33250]: osdmap e35: 6 total, 5 up, 6 in 2024-05-03T17:16:50.408 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:50.408 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:50 smithi070 ceph-mon[33250]: from='osd.5 [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:16:50.408 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:50 smithi070 ceph-mon[33250]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:16:51.242 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:51 smithi082 ceph-mon[33535]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T17:16:51.242 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:51 smithi082 ceph-mon[33535]: osdmap e36: 6 total, 5 up, 6 in 2024-05-03T17:16:51.242 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:51 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:51.242 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:51 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:51.242 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:51 smithi082 ceph-mon[33535]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:51.294 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:51 smithi067 ceph-mon[29731]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T17:16:51.294 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:51 smithi067 ceph-mon[29731]: osdmap e36: 6 total, 5 up, 6 in 2024-05-03T17:16:51.294 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:51 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:51.294 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:51 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:51.294 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:51 smithi067 ceph-mon[29731]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:51.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:51 smithi070 ceph-mon[33250]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T17:16:51.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:51 smithi070 ceph-mon[33250]: osdmap e36: 6 total, 5 up, 6 in 2024-05-03T17:16:51.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:51 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:51.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:51 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:51.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:51 smithi070 ceph-mon[33250]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-03T17:16:52.026 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-03T17:16:52.026 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-03T17:16:52.026 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-05-03T17:16:52.026 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-05-03T17:16:52.026 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124717 s, 841 MB/s 2024-05-03T17:16:52.026 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-05-03T17:16:52.339 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:52 smithi067 ceph-mon[29731]: purged_snaps scrub starts 2024-05-03T17:16:52.339 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:52 smithi067 ceph-mon[29731]: purged_snaps scrub ok 2024-05-03T17:16:52.339 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:52 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:52.339 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:52 smithi067 ceph-mon[29731]: osd.5 [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290] boot 2024-05-03T17:16:52.339 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:52 smithi067 ceph-mon[29731]: osdmap e37: 6 total, 6 up, 6 in 2024-05-03T17:16:52.339 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:52 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:52.339 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:52 smithi067 ceph-mon[29731]: osdmap e38: 6 total, 6 up, 6 in 2024-05-03T17:16:52.432 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:52 smithi082 ceph-mon[33535]: purged_snaps scrub starts 2024-05-03T17:16:52.432 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:52 smithi082 ceph-mon[33535]: purged_snaps scrub ok 2024-05-03T17:16:52.432 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:52 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:52.432 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:52 smithi082 ceph-mon[33535]: osd.5 [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290] boot 2024-05-03T17:16:52.433 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:52 smithi082 ceph-mon[33535]: osdmap e37: 6 total, 6 up, 6 in 2024-05-03T17:16:52.433 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:52 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:52.433 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:52 smithi082 ceph-mon[33535]: osdmap e38: 6 total, 6 up, 6 in 2024-05-03T17:16:52.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:52 smithi070 ceph-mon[33250]: purged_snaps scrub starts 2024-05-03T17:16:52.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:52 smithi070 ceph-mon[33250]: purged_snaps scrub ok 2024-05-03T17:16:52.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:52 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:52.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:52 smithi070 ceph-mon[33250]: osd.5 [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290] boot 2024-05-03T17:16:52.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:52 smithi070 ceph-mon[33250]: osdmap e37: 6 total, 6 up, 6 in 2024-05-03T17:16:52.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:52 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-03T17:16:52.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:52 smithi070 ceph-mon[33250]: osdmap e38: 6 total, 6 up, 6 in 2024-05-03T17:16:52.834 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_2 2024-05-03T17:16:53.088 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:16:54.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:53 smithi070 ceph-mon[33250]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:16:54.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:53 smithi070 ceph-mon[33250]: Detected new or changed devices on smithi082 2024-05-03T17:16:54.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:54.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:54.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:54.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:54.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:53 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi082 to 8544M 2024-05-03T17:16:54.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:54.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:54.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:54.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:54.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:53 smithi070 ceph-mon[33250]: osdmap e39: 6 total, 6 up, 6 in 2024-05-03T17:16:54.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:53 smithi067 ceph-mon[29731]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:16:54.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:53 smithi067 ceph-mon[29731]: Detected new or changed devices on smithi082 2024-05-03T17:16:54.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:54.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:54.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:54.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:54.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:53 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi082 to 8544M 2024-05-03T17:16:54.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:54.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:54.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:54.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:54.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:53 smithi067 ceph-mon[29731]: osdmap e39: 6 total, 6 up, 6 in 2024-05-03T17:16:54.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:53 smithi082 ceph-mon[33535]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:16:54.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:53 smithi082 ceph-mon[33535]: Detected new or changed devices on smithi082 2024-05-03T17:16:54.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:54.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:54.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:54.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:16:54.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:53 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi082 to 8544M 2024-05-03T17:16:54.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:54.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:54.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:16:54.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:16:54.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:53 smithi082 ceph-mon[33535]: osdmap e39: 6 total, 6 up, 6 in 2024-05-03T17:16:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.790+0000 7f4ec18d6700 1 -- 172.21.15.67:0/1042019608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ebc1091f0 msgr2=0x7f4ebc10b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.790+0000 7f4ec18d6700 1 --2- 172.21.15.67:0/1042019608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ebc1091f0 0x7f4ebc10b660 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f4eac009990 tx=0x7f4eac009ca0 comp rx=0 tx=0).stop 2024-05-03T17:16:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.790+0000 7f4ec18d6700 1 -- 172.21.15.67:0/1042019608 shutdown_connections 2024-05-03T17:16:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.790+0000 7f4ec18d6700 1 --2- 172.21.15.67:0/1042019608 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ebc1091f0 0x7f4ebc10b660 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.790+0000 7f4ec18d6700 1 --2- 172.21.15.67:0/1042019608 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ebc103f90 0x7f4ebc106380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.790+0000 7f4ec18d6700 1 --2- 172.21.15.67:0/1042019608 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ebc1068c0 0x7f4ebc108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.790+0000 7f4ec18d6700 1 -- 172.21.15.67:0/1042019608 >> 172.21.15.67:0/1042019608 conn(0x7f4ebc0fd9a0 msgr2=0x7f4ebc0ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:16:54.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.790+0000 7f4ec18d6700 1 -- 172.21.15.67:0/1042019608 shutdown_connections 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.790+0000 7f4ec18d6700 1 -- 172.21.15.67:0/1042019608 wait complete. 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.791+0000 7f4ec18d6700 1 Processor -- start 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.791+0000 7f4ec18d6700 1 -- start start 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.791+0000 7f4ec18d6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ebc103f90 0x7f4ebc1038f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.791+0000 7f4ec18d6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ebc1068c0 0x7f4ebc101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.791+0000 7f4ec18d6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ebc1091f0 0x7f4ebc102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.791+0000 7f4ec18d6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc10f010 con 0x7f4ebc1091f0 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.791+0000 7f4ec18d6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc10ee90 con 0x7f4ebc103f90 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.791+0000 7f4ec18d6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4ebc10f190 con 0x7f4ebc1068c0 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.791+0000 7f4ebb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ebc1091f0 0x7f4ebc102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.791+0000 7f4ebb7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ebc1091f0 0x7f4ebc102480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39600/0 (socket says 172.21.15.67:39600) 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.791+0000 7f4ebb7fe700 1 -- 172.21.15.67:0/124355870 learned_addr learned my addr 172.21.15.67:0/124355870 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4eba7fc700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ebc1068c0 0x7f4ebc101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:16:54.794 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4ebaffd700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ebc103f90 0x7f4ebc1038f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:16:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4ebb7fe700 1 -- 172.21.15.67:0/124355870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ebc1068c0 msgr2=0x7f4ebc101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4ebb7fe700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ebc1068c0 0x7f4ebc101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4ebb7fe700 1 -- 172.21.15.67:0/124355870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ebc103f90 msgr2=0x7f4ebc1038f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:16:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4ebb7fe700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ebc103f90 0x7f4ebc1038f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:16:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4ebb7fe700 1 -- 172.21.15.67:0/124355870 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ebc102d30 con 0x7f4ebc1091f0 2024-05-03T17:16:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4ebb7fe700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ebc1091f0 0x7f4ebc102480 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4eac005b30 tx=0x7f4eac009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:16:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4ec08d4700 1 -- 172.21.15.67:0/124355870 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac00fb80 con 0x7f4ebc1091f0 2024-05-03T17:16:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4ec18d6700 1 -- 172.21.15.67:0/124355870 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ebc1a3770 con 0x7f4ebc1091f0 2024-05-03T17:16:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4ec08d4700 1 -- 172.21.15.67:0/124355870 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4eac00fd20 con 0x7f4ebc1091f0 2024-05-03T17:16:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4ec08d4700 1 -- 172.21.15.67:0/124355870 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eac005670 con 0x7f4ebc1091f0 2024-05-03T17:16:54.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.792+0000 7f4ec18d6700 1 -- 172.21.15.67:0/124355870 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ebc1a3b60 con 0x7f4ebc1091f0 2024-05-03T17:16:54.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.793+0000 7f4ec18d6700 1 -- 172.21.15.67:0/124355870 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ebc06a190 con 0x7f4ebc1091f0 2024-05-03T17:16:54.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.794+0000 7f4ec08d4700 1 -- 172.21.15.67:0/124355870 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f4eac009d10 con 0x7f4ebc1091f0 2024-05-03T17:16:54.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.795+0000 7f4ec08d4700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4ea80b2880 0x7f4ea80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:16:54.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.795+0000 7f4ebaffd700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4ea80b2880 0x7f4ea80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:16:54.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.795+0000 7f4ec08d4700 1 -- 172.21.15.67:0/124355870 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4533+0+0 (secure 0 0 0) 0x7f4eac0e25c0 con 0x7f4ebc1091f0 2024-05-03T17:16:54.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.796+0000 7f4ebaffd700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4ea80b2880 0x7f4ea80b4d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009380 tx=0x7f4eb400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:16:54.799 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:54.799+0000 7f4ec08d4700 1 -- 172.21.15.67:0/124355870 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4eac013070 con 0x7f4ebc1091f0 2024-05-03T17:16:55.024 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:16:55.024+0000 7f4ec18d6700 1 -- 172.21.15.67:0/124355870 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4ebc0639a0 con 0x7f4ea80b2880 2024-05-03T17:16:55.939 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:55 smithi067 ceph-mon[29731]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:16:55.939 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:55 smithi067 ceph-mon[29731]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:16:55.939 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:16:55.939 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:16:55.939 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:56.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:55 smithi082 ceph-mon[33535]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:16:56.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:55 smithi082 ceph-mon[33535]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:16:56.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:16:56.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:16:56.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:56.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:55 smithi070 ceph-mon[33250]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:16:56.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:55 smithi070 ceph-mon[33250]: from='client.14349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:16:56.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:16:56.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:16:56.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:16:58.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:57 smithi067 ceph-mon[29731]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:16:58.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:57 smithi082 ceph-mon[33535]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:16:58.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:57 smithi070 ceph-mon[33250]: pgmap v144: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:16:59.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:58 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2079859784' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d8a5e8a-66ac-462e-b960-24c0ee248fac"}]: dispatch 2024-05-03T17:16:59.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:58 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2079859784' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d8a5e8a-66ac-462e-b960-24c0ee248fac"}]': finished 2024-05-03T17:16:59.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:58 smithi067 ceph-mon[29731]: osdmap e40: 7 total, 6 up, 7 in 2024-05-03T17:16:59.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:16:59.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:58 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2800946644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:16:59.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:58 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2079859784' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d8a5e8a-66ac-462e-b960-24c0ee248fac"}]: dispatch 2024-05-03T17:16:59.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:58 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2079859784' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d8a5e8a-66ac-462e-b960-24c0ee248fac"}]': finished 2024-05-03T17:16:59.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:58 smithi082 ceph-mon[33535]: osdmap e40: 7 total, 6 up, 7 in 2024-05-03T17:16:59.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:16:59.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:58 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2800946644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:16:59.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:58 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2079859784' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4d8a5e8a-66ac-462e-b960-24c0ee248fac"}]: dispatch 2024-05-03T17:16:59.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:58 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2079859784' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4d8a5e8a-66ac-462e-b960-24c0ee248fac"}]': finished 2024-05-03T17:16:59.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:58 smithi070 ceph-mon[33250]: osdmap e40: 7 total, 6 up, 7 in 2024-05-03T17:16:59.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:58 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:16:59.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:58 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2800946644' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:17:00.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:16:59 smithi082 ceph-mon[33535]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-03T17:17:00.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:16:59 smithi070 ceph-mon[33250]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-03T17:17:00.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:16:59 smithi067 ceph-mon[29731]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-05-03T17:17:02.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:01 smithi082 ceph-mon[33535]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-03T17:17:02.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:01 smithi070 ceph-mon[33250]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-03T17:17:02.328 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:01 smithi067 ceph-mon[29731]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-03T17:17:04.237 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:03 smithi067 ceph-mon[29731]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-03T17:17:04.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:03 smithi082 ceph-mon[33535]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-03T17:17:04.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:03 smithi070 ceph-mon[33250]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-03T17:17:06.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:05 smithi082 ceph-mon[33535]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:17:06.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-03T17:17:06.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:06.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:05 smithi070 ceph-mon[33250]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:17:06.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-03T17:17:06.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:06.309 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:05 smithi067 ceph-mon[29731]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:17:06.309 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-03T17:17:06.310 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:07.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:06 smithi082 ceph-mon[33535]: Deploying daemon osd.6 on smithi067 2024-05-03T17:17:07.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:06 smithi070 ceph-mon[33250]: Deploying daemon osd.6 on smithi067 2024-05-03T17:17:07.358 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:06 smithi067 ceph-mon[29731]: Deploying daemon osd.6 on smithi067 2024-05-03T17:17:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:07 smithi082 ceph-mon[33535]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:17:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:07 smithi070 ceph-mon[33250]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:17:08.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:07 smithi067 ceph-mon[29731]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:17:10.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:09 smithi067 ceph-mon[29731]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:10.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:09 smithi082 ceph-mon[33535]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:10.310 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:09 smithi070 ceph-mon[33250]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:12.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:11 smithi082 ceph-mon[33535]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:12.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:11 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:12.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:11 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:12.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:11 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:17:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:11 smithi070 ceph-mon[33250]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:11 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:11 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:11 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:17:12.439 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:11 smithi067 ceph-mon[29731]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:12.439 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:11 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:12.439 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:11 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:12.439 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:11 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:17:14.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:13 smithi082 ceph-mon[33535]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:14.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:13 smithi070 ceph-mon[33250]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:14.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:13 smithi067 ceph-mon[29731]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:16.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.083+0000 7f4ec08d4700 1 -- 172.21.15.67:0/124355870 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4ebc0639a0 con 0x7f4ea80b2880 2024-05-03T17:17:16.084 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 6 on host 'smithi067' 2024-05-03T17:17:16.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.085+0000 7f4ec18d6700 1 -- 172.21.15.67:0/124355870 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4ea80b2880 msgr2=0x7f4ea80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:17:16.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.085+0000 7f4ec18d6700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4ea80b2880 0x7f4ea80b4d40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009380 tx=0x7f4eb400b040 comp rx=0 tx=0).stop 2024-05-03T17:17:16.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.085+0000 7f4ec18d6700 1 -- 172.21.15.67:0/124355870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ebc1091f0 msgr2=0x7f4ebc102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:17:16.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.085+0000 7f4ec18d6700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ebc1091f0 0x7f4ebc102480 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4eac005b30 tx=0x7f4eac009510 comp rx=0 tx=0).stop 2024-05-03T17:17:16.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.085+0000 7f4ec18d6700 1 -- 172.21.15.67:0/124355870 shutdown_connections 2024-05-03T17:17:16.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.085+0000 7f4ec18d6700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4ea80b2880 0x7f4ea80b4d40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:16.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.085+0000 7f4ec18d6700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4ebc1091f0 0x7f4ebc102480 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:16.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.085+0000 7f4ec18d6700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4ebc103f90 0x7f4ebc1038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:16.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.085+0000 7f4ec18d6700 1 --2- 172.21.15.67:0/124355870 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4ebc1068c0 0x7f4ebc101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:16.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.085+0000 7f4ec18d6700 1 -- 172.21.15.67:0/124355870 >> 172.21.15.67:0/124355870 conn(0x7f4ebc0fd9a0 msgr2=0x7f4ebc104830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:17:16.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.086+0000 7f4ec18d6700 1 -- 172.21.15.67:0/124355870 shutdown_connections 2024-05-03T17:17:16.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:17:16.086+0000 7f4ec18d6700 1 -- 172.21.15.67:0/124355870 wait complete. 2024-05-03T17:17:16.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:15 smithi082 ceph-mon[33535]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:16.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:15 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:16.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:15 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:16.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:15 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:16.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:15 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:17:16.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:15 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:16.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:15 smithi082 ceph-mon[33535]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-03T17:17:16.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:15 smithi070 ceph-mon[33250]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:16.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:15 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:16.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:15 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:16.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:15 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:16.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:15 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:17:16.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:15 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:16.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:15 smithi070 ceph-mon[33250]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-03T17:17:16.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:15 smithi067 ceph-mon[29731]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:16.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:15 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:16.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:15 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:16.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:15 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:16.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:15 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:17:16.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:15 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:16.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:15 smithi067 ceph-mon[29731]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-03T17:17:16.953 DEBUG:teuthology.orchestra.run.smithi067:osd.6> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.6.service 2024-05-03T17:17:16.955 INFO:tasks.cephadm:Deploying osd.7 on smithi070 with /dev/vg_nvme/lv_2... 2024-05-03T17:17:16.955 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-03T17:17:17.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:17 smithi067 ceph-mon[29731]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-03T17:17:17.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:17 smithi067 ceph-mon[29731]: osdmap e41: 7 total, 6 up, 7 in 2024-05-03T17:17:17.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:17 smithi067 ceph-mon[29731]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:17:17.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:17.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:17.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:17.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:17:17.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:17 smithi067 ceph-mon[29731]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:17.206 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T17:17:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:17 smithi070 ceph-mon[33250]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-03T17:17:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:17 smithi070 ceph-mon[33250]: osdmap e41: 7 total, 6 up, 7 in 2024-05-03T17:17:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:17 smithi070 ceph-mon[33250]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:17:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:17:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:17 smithi070 ceph-mon[33250]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:17.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:17 smithi082 ceph-mon[33535]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-03T17:17:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:17 smithi082 ceph-mon[33535]: osdmap e41: 7 total, 6 up, 7 in 2024-05-03T17:17:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:17 smithi082 ceph-mon[33535]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:17:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:17:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:17 smithi082 ceph-mon[33535]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:17.703 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:17:17 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:17:17.491+0000 7f3f227f0700 -1 osd.6 0 waiting for initial osdmap 2024-05-03T17:17:17.704 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:17:17 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:17:17.500+0000 7f3f1a5db700 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T17:17:18.346 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:18 smithi070 ceph-mon[33250]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T17:17:18.346 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:18 smithi070 ceph-mon[33250]: osdmap e42: 7 total, 6 up, 7 in 2024-05-03T17:17:18.346 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:18 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:18.346 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:18 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:18.346 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:18 smithi070 ceph-mon[33250]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' 2024-05-03T17:17:18.416 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:18 smithi082 ceph-mon[33535]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T17:17:18.416 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:18 smithi082 ceph-mon[33535]: osdmap e42: 7 total, 6 up, 7 in 2024-05-03T17:17:18.416 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:18 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:18.416 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:18 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:18.416 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:18 smithi082 ceph-mon[33535]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' 2024-05-03T17:17:18.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:18 smithi067 ceph-mon[29731]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T17:17:18.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:18 smithi067 ceph-mon[29731]: osdmap e42: 7 total, 6 up, 7 in 2024-05-03T17:17:18.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:18 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:18.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:18 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:18.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:18 smithi067 ceph-mon[29731]: from='osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161]' entity='osd.6' 2024-05-03T17:17:19.110 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-03T17:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-03T17:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-05-03T17:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-05-03T17:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00842261 s, 1.2 GB/s 2024-05-03T17:17:19.111 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-05-03T17:17:19.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:19 smithi067 ceph-mon[29731]: purged_snaps scrub starts 2024-05-03T17:17:19.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:19 smithi067 ceph-mon[29731]: purged_snaps scrub ok 2024-05-03T17:17:19.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:19 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:19.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:19 smithi067 ceph-mon[29731]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:19.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:19 smithi067 ceph-mon[29731]: osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161] boot 2024-05-03T17:17:19.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:19 smithi067 ceph-mon[29731]: osdmap e43: 7 total, 7 up, 7 in 2024-05-03T17:17:19.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:19 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:19.438 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:19 smithi070 ceph-mon[33250]: purged_snaps scrub starts 2024-05-03T17:17:19.438 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:19 smithi070 ceph-mon[33250]: purged_snaps scrub ok 2024-05-03T17:17:19.438 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:19 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:19.438 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:19 smithi070 ceph-mon[33250]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:19.439 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:19 smithi070 ceph-mon[33250]: osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161] boot 2024-05-03T17:17:19.439 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:19 smithi070 ceph-mon[33250]: osdmap e43: 7 total, 7 up, 7 in 2024-05-03T17:17:19.439 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:19 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:19.439 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:19 smithi082 ceph-mon[33535]: purged_snaps scrub starts 2024-05-03T17:17:19.440 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:19 smithi082 ceph-mon[33535]: purged_snaps scrub ok 2024-05-03T17:17:19.440 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:19 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:19.440 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:19 smithi082 ceph-mon[33535]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-05-03T17:17:19.440 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:19 smithi082 ceph-mon[33535]: osd.6 [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161] boot 2024-05-03T17:17:19.440 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:19 smithi082 ceph-mon[33535]: osdmap e43: 7 total, 7 up, 7 in 2024-05-03T17:17:19.440 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:19 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-03T17:17:19.776 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_2 2024-05-03T17:17:20.019 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T17:17:20.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:20 smithi082 ceph-mon[33535]: osdmap e44: 7 total, 7 up, 7 in 2024-05-03T17:17:20.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:20 smithi070 ceph-mon[33250]: osdmap e44: 7 total, 7 up, 7 in 2024-05-03T17:17:20.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:20 smithi067 ceph-mon[29731]: osdmap e44: 7 total, 7 up, 7 in 2024-05-03T17:17:21.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.594+0000 7f35a9ea5700 1 -- 172.21.15.70:0/1984086487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35a41050a0 msgr2=0x7f35a410b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:17:21.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.594+0000 7f35a9ea5700 1 --2- 172.21.15.70:0/1984086487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35a41050a0 0x7f35a410b950 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f359c00b0d0 tx=0x7f359c00b3e0 comp rx=0 tx=0).stop 2024-05-03T17:17:21.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.594+0000 7f35a9ea5700 1 -- 172.21.15.70:0/1984086487 shutdown_connections 2024-05-03T17:17:21.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.594+0000 7f35a9ea5700 1 --2- 172.21.15.70:0/1984086487 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35a41050a0 0x7f35a410b950 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:21.598 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.594+0000 7f35a9ea5700 1 --2- 172.21.15.70:0/1984086487 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a4103500 0x7f35a4103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:21.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.594+0000 7f35a9ea5700 1 --2- 172.21.15.70:0/1984086487 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35a4104700 0x7f35a4104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:21.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.594+0000 7f35a9ea5700 1 -- 172.21.15.70:0/1984086487 >> 172.21.15.70:0/1984086487 conn(0x7f35a40feaa0 msgr2=0x7f35a4100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:17:21.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.594+0000 7f35a9ea5700 1 -- 172.21.15.70:0/1984086487 shutdown_connections 2024-05-03T17:17:21.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.594+0000 7f35a9ea5700 1 -- 172.21.15.70:0/1984086487 wait complete. 2024-05-03T17:17:21.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.595+0000 7f35a9ea5700 1 Processor -- start 2024-05-03T17:17:21.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.595+0000 7f35a9ea5700 1 -- start start 2024-05-03T17:17:21.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.595+0000 7f35a9ea5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35a4103500 0x7f35a4199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:17:21.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.595+0000 7f35a9ea5700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35a4104700 0x7f35a4199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:17:21.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.595+0000 7f35a9ea5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a41050a0 0x7f35a41a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:17:21.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.595+0000 7f35a9ea5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f35a410eff0 con 0x7f35a41050a0 2024-05-03T17:17:21.599 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.595+0000 7f35a9ea5700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35a410ee70 con 0x7f35a4103500 2024-05-03T17:17:21.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.595+0000 7f35a9ea5700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f35a410f170 con 0x7f35a4104700 2024-05-03T17:17:21.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35a4103500 0x7f35a4199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:17:21.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a37fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35a4103500 0x7f35a4199610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:56980/0 (socket says 172.21.15.70:56980) 2024-05-03T17:17:21.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a37fe700 1 -- 172.21.15.70:0/3685926733 learned_addr learned my addr 172.21.15.70:0/3685926733 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-05-03T17:17:21.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a2ffd700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35a4104700 0x7f35a4199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:17:21.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a3fff700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a41050a0 0x7f35a41a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:17:21.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a37fe700 1 -- 172.21.15.70:0/3685926733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35a4104700 msgr2=0x7f35a4199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:17:21.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a37fe700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35a4104700 0x7f35a4199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:21.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a37fe700 1 -- 172.21.15.70:0/3685926733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a41050a0 msgr2=0x7f35a41a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:17:21.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a37fe700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a41050a0 0x7f35a41a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:21.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a37fe700 1 -- 172.21.15.70:0/3685926733 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35a41a1270 con 0x7f35a4103500 2024-05-03T17:17:21.600 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a3fff700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a41050a0 0x7f35a41a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:17:21.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a2ffd700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35a4104700 0x7f35a4199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:17:21.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a37fe700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35a4103500 0x7f35a4199610 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f359400d980 tx=0x7f359400dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:17:21.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a0ff9700 1 -- 172.21.15.70:0/3685926733 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3594016070 con 0x7f35a4103500 2024-05-03T17:17:21.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a0ff9700 1 -- 172.21.15.70:0/3685926733 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f359400bb60 con 0x7f35a4103500 2024-05-03T17:17:21.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a0ff9700 1 -- 172.21.15.70:0/3685926733 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359400f3c0 con 0x7f35a4103500 2024-05-03T17:17:21.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a9ea5700 1 -- 172.21.15.70:0/3685926733 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35a41a1560 con 0x7f35a4103500 2024-05-03T17:17:21.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.596+0000 7f35a9ea5700 1 -- 172.21.15.70:0/3685926733 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35a41a1a50 con 0x7f35a4103500 2024-05-03T17:17:21.601 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.597+0000 7f35a9ea5700 1 -- 172.21.15.70:0/3685926733 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35a406a190 con 0x7f35a4103500 2024-05-03T17:17:21.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.598+0000 7f35a0ff9700 1 -- 172.21.15.70:0/3685926733 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f3594019070 con 0x7f35a4103500 2024-05-03T17:17:21.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.599+0000 7f35a0ff9700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f358c0b2900 0x7f358c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:17:21.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.599+0000 7f35a0ff9700 1 -- 172.21.15.70:0/3685926733 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4965+0+0 (secure 0 0 0) 0x7f35940e1110 con 0x7f35a4103500 2024-05-03T17:17:21.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.599+0000 7f35a2ffd700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f358c0b2900 0x7f358c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:17:21.602 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.600+0000 7f35a2ffd700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f358c0b2900 0x7f358c0b4dc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f35980099d0 tx=0x7f3598008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:17:21.604 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.604+0000 7f35a0ff9700 1 -- 172.21.15.70:0/3685926733 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f3594020030 con 0x7f35a4103500 2024-05-03T17:17:21.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:21 smithi070 ceph-mon[33250]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:21.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:21 smithi070 ceph-mon[33250]: Detected new or changed devices on smithi067 2024-05-03T17:17:21.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:21.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:21.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:21.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:21.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:21.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:21 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi067 to 5696M 2024-05-03T17:17:21.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:21.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:21.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:17:21.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:21.817 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:21.817+0000 7f35a9ea5700 1 -- 172.21.15.70:0/3685926733 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f35a40638e0 con 0x7f358c0b2900 2024-05-03T17:17:21.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:21 smithi067 ceph-mon[29731]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:21.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:21 smithi067 ceph-mon[29731]: Detected new or changed devices on smithi067 2024-05-03T17:17:21.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:21.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:21.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:21.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:21.856 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:21.856 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:21 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi067 to 5696M 2024-05-03T17:17:21.856 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:21.856 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:21.856 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:17:21.856 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:22.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:21 smithi082 ceph-mon[33535]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:22.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:21 smithi082 ceph-mon[33535]: Detected new or changed devices on smithi067 2024-05-03T17:17:22.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:22.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:22.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:22.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:22.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:22.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:21 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi067 to 5696M 2024-05-03T17:17:22.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:22.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:22.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:17:22.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:22.752 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:22 smithi070 ceph-mon[33250]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:17:22.752 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:22 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:17:22.752 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:22 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:17:22.752 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:22 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:22.867 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:22 smithi067 ceph-mon[29731]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:17:22.867 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:22 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:17:22.867 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:22 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:17:22.867 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:22 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:23.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:22 smithi082 ceph-mon[33535]: from='client.24248 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:17:23.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:22 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:17:23.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:22 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:17:23.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:22 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:23.726 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:23 smithi070 ceph-mon[33250]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:23.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:23 smithi067 ceph-mon[29731]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:24.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:23 smithi082 ceph-mon[33535]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:25.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:25 smithi070 ceph-mon[33250]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:25.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:25 smithi070 ceph-mon[33250]: from='client.? 172.21.15.70:0/1108636454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb983837-8172-4802-9843-6360cda8f4be"}]: dispatch 2024-05-03T17:17:25.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:25 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb983837-8172-4802-9843-6360cda8f4be"}]: dispatch 2024-05-03T17:17:25.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:25 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb983837-8172-4802-9843-6360cda8f4be"}]': finished 2024-05-03T17:17:25.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:25 smithi070 ceph-mon[33250]: osdmap e45: 8 total, 7 up, 8 in 2024-05-03T17:17:25.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:25.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:25 smithi070 ceph-mon[33250]: from='client.? 172.21.15.70:0/1519518917' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:17:25.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:25 smithi067 ceph-mon[29731]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:25.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:25 smithi067 ceph-mon[29731]: from='client.? 172.21.15.70:0/1108636454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb983837-8172-4802-9843-6360cda8f4be"}]: dispatch 2024-05-03T17:17:25.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:25 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb983837-8172-4802-9843-6360cda8f4be"}]: dispatch 2024-05-03T17:17:25.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:25 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb983837-8172-4802-9843-6360cda8f4be"}]': finished 2024-05-03T17:17:25.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:25 smithi067 ceph-mon[29731]: osdmap e45: 8 total, 7 up, 8 in 2024-05-03T17:17:25.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:25.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:25 smithi067 ceph-mon[29731]: from='client.? 172.21.15.70:0/1519518917' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:17:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:25 smithi082 ceph-mon[33535]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:25 smithi082 ceph-mon[33535]: from='client.? 172.21.15.70:0/1108636454' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb983837-8172-4802-9843-6360cda8f4be"}]: dispatch 2024-05-03T17:17:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:25 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "eb983837-8172-4802-9843-6360cda8f4be"}]: dispatch 2024-05-03T17:17:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:25 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "eb983837-8172-4802-9843-6360cda8f4be"}]': finished 2024-05-03T17:17:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:25 smithi082 ceph-mon[33535]: osdmap e45: 8 total, 7 up, 8 in 2024-05-03T17:17:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:26.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:25 smithi082 ceph-mon[33535]: from='client.? 172.21.15.70:0/1519518917' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:17:27.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:27 smithi070 ceph-mon[33250]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:27.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:27 smithi067 ceph-mon[29731]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:28.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:27 smithi082 ceph-mon[33535]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:29.742 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:29 smithi070 ceph-mon[33250]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:29.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:29 smithi067 ceph-mon[29731]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:30.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:29 smithi082 ceph-mon[33535]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:31.644 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:31 smithi070 ceph-mon[33250]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:31 smithi067 ceph-mon[29731]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:32.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:31 smithi082 ceph-mon[33535]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:32.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:32 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-03T17:17:32.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:32 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:32.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:32 smithi067 ceph-mon[29731]: Deploying daemon osd.7 on smithi070 2024-05-03T17:17:32.984 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:32 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-03T17:17:32.984 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:32 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:32.984 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:32 smithi070 ceph-mon[33250]: Deploying daemon osd.7 on smithi070 2024-05-03T17:17:33.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:32 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-03T17:17:33.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:32 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:33.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:32 smithi082 ceph-mon[33535]: Deploying daemon osd.7 on smithi070 2024-05-03T17:17:33.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:33 smithi067 ceph-mon[29731]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:33.957 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:33 smithi070 ceph-mon[33250]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:34.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:33 smithi082 ceph-mon[33535]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:35.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:35 smithi070 ceph-mon[33250]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:35.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:35 smithi067 ceph-mon[29731]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:36.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:35 smithi082 ceph-mon[33535]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:37.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:37 smithi070 ceph-mon[33250]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:37.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:37 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:37.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:37 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:37.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:37 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:17:37.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:37 smithi067 ceph-mon[29731]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:37.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:37 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:37.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:37 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:37.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:37 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:17:38.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:37 smithi082 ceph-mon[33535]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:38.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:37 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:38.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:37 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:38.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:37 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:17:39.791 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:39 smithi070 ceph-mon[33250]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:39.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:39 smithi067 ceph-mon[29731]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:40.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:39 smithi082 ceph-mon[33535]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:40.783 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.782+0000 7f35a0ff9700 1 -- 172.21.15.70:0/3685926733 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f35a40638e0 con 0x7f358c0b2900 2024-05-03T17:17:40.783 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 7 on host 'smithi070' 2024-05-03T17:17:40.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.784+0000 7f35a9ea5700 1 -- 172.21.15.70:0/3685926733 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f358c0b2900 msgr2=0x7f358c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:17:40.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.784+0000 7f35a9ea5700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f358c0b2900 0x7f358c0b4dc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f35980099d0 tx=0x7f3598008040 comp rx=0 tx=0).stop 2024-05-03T17:17:40.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.784+0000 7f35a9ea5700 1 -- 172.21.15.70:0/3685926733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35a4103500 msgr2=0x7f35a4199610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:17:40.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.784+0000 7f35a9ea5700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35a4103500 0x7f35a4199610 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f359400d980 tx=0x7f359400dd40 comp rx=0 tx=0).stop 2024-05-03T17:17:40.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.784+0000 7f35a9ea5700 1 -- 172.21.15.70:0/3685926733 shutdown_connections 2024-05-03T17:17:40.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.784+0000 7f35a9ea5700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f358c0b2900 0x7f358c0b4dc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:40.785 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.784+0000 7f35a9ea5700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35a41050a0 0x7f35a41a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.784+0000 7f35a9ea5700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35a4103500 0x7f35a4199610 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.784+0000 7f35a9ea5700 1 --2- 172.21.15.70:0/3685926733 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35a4104700 0x7f35a4199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.784+0000 7f35a9ea5700 1 -- 172.21.15.70:0/3685926733 >> 172.21.15.70:0/3685926733 conn(0x7f35a40feaa0 msgr2=0x7f35a4107a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:17:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.785+0000 7f35a9ea5700 1 -- 172.21.15.70:0/3685926733 shutdown_connections 2024-05-03T17:17:40.786 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:17:40.785+0000 7f35a9ea5700 1 -- 172.21.15.70:0/3685926733 wait complete. 2024-05-03T17:17:41.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:40 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:40 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:40 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:41.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:40 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:17:41.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:40 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:40 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:40 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:17:41.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:40 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:40 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:40 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:40 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:41.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:40 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:17:41.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:40 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:40 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:40 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:17:41.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:40 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:40 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:40 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:40 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:41.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:40 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:17:41.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:40 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:40 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:40 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:17:41.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:40 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:41.434 DEBUG:teuthology.orchestra.run.smithi070:osd.7> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.7.service 2024-05-03T17:17:41.436 INFO:tasks.cephadm:Deploying osd.8 on smithi082 with /dev/vg_nvme/lv_2... 2024-05-03T17:17:41.437 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-03T17:17:41.682 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:17:42.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:41 smithi070 ceph-mon[33250]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:42.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:41 smithi070 ceph-mon[33250]: from='osd.7 [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T17:17:42.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:41 smithi070 ceph-mon[33250]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T17:17:42.122 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:41 smithi082 ceph-mon[33535]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:42.122 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:41 smithi082 ceph-mon[33535]: from='osd.7 [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T17:17:42.122 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:41 smithi082 ceph-mon[33535]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T17:17:42.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:41 smithi067 ceph-mon[29731]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:42.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:41 smithi067 ceph-mon[29731]: from='osd.7 [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T17:17:42.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:41 smithi067 ceph-mon[29731]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-03T17:17:43.081 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:42 smithi082 ceph-mon[33535]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-03T17:17:43.081 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:42 smithi082 ceph-mon[33535]: from='osd.7 [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:17:43.081 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:42 smithi082 ceph-mon[33535]: osdmap e46: 8 total, 7 up, 8 in 2024-05-03T17:17:43.081 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:42 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:43.081 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:42 smithi082 ceph-mon[33535]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:17:43.082 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:42 smithi082 ceph-mon[33535]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-05-03T17:17:43.082 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:42 smithi082 ceph-mon[33535]: osdmap e47: 8 total, 7 up, 8 in 2024-05-03T17:17:43.082 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:42 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:43.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:42 smithi070 ceph-mon[33250]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-03T17:17:43.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:42 smithi070 ceph-mon[33250]: from='osd.7 [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:17:43.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:42 smithi070 ceph-mon[33250]: osdmap e46: 8 total, 7 up, 8 in 2024-05-03T17:17:43.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:42 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:43.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:42 smithi070 ceph-mon[33250]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:17:43.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:42 smithi070 ceph-mon[33250]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-05-03T17:17:43.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:42 smithi070 ceph-mon[33250]: osdmap e47: 8 total, 7 up, 8 in 2024-05-03T17:17:43.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:42 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:43.309 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:17:42 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:17:42.967+0000 7fd8b472b700 -1 osd.7 0 waiting for initial osdmap 2024-05-03T17:17:43.309 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:17:42 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:17:42.977+0000 7fd8b051e700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T17:17:43.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:42 smithi067 ceph-mon[29731]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-03T17:17:43.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:42 smithi067 ceph-mon[29731]: from='osd.7 [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:17:43.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:42 smithi067 ceph-mon[29731]: osdmap e46: 8 total, 7 up, 8 in 2024-05-03T17:17:43.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:42 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:43.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:42 smithi067 ceph-mon[29731]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:17:43.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:42 smithi067 ceph-mon[29731]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-05-03T17:17:43.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:42 smithi067 ceph-mon[29731]: osdmap e47: 8 total, 7 up, 8 in 2024-05-03T17:17:43.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:42 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:43.650 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-03T17:17:43.650 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-03T17:17:43.650 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-05-03T17:17:43.650 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-05-03T17:17:43.650 INFO:teuthology.orchestra.run.smithi082.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122515 s, 856 MB/s 2024-05-03T17:17:43.650 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-05-03T17:17:44.256 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:43 smithi070 ceph-mon[33250]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:44.256 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:44.256 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:43 smithi070 ceph-mon[33250]: osd.7 [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173] boot 2024-05-03T17:17:44.256 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:43 smithi070 ceph-mon[33250]: osdmap e48: 8 total, 8 up, 8 in 2024-05-03T17:17:44.256 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:44.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:43 smithi082 ceph-mon[33535]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:44.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:44.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:43 smithi082 ceph-mon[33535]: osd.7 [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173] boot 2024-05-03T17:17:44.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:43 smithi082 ceph-mon[33535]: osdmap e48: 8 total, 8 up, 8 in 2024-05-03T17:17:44.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:44.293 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_2 2024-05-03T17:17:44.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:43 smithi067 ceph-mon[29731]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-03T17:17:44.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:44.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:43 smithi067 ceph-mon[29731]: osd.7 [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173] boot 2024-05-03T17:17:44.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:43 smithi067 ceph-mon[29731]: osdmap e48: 8 total, 8 up, 8 in 2024-05-03T17:17:44.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-03T17:17:44.536 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:17:45.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:44 smithi082 ceph-mon[33535]: purged_snaps scrub starts 2024-05-03T17:17:45.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:44 smithi082 ceph-mon[33535]: purged_snaps scrub ok 2024-05-03T17:17:45.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:45.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:45.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:45.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:45.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:45.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:45.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:45.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:17:45.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:45.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:44 smithi082 ceph-mon[33535]: osdmap e49: 8 total, 8 up, 8 in 2024-05-03T17:17:45.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:44 smithi070 ceph-mon[33250]: purged_snaps scrub starts 2024-05-03T17:17:45.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:44 smithi070 ceph-mon[33250]: purged_snaps scrub ok 2024-05-03T17:17:45.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:45.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:45.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:45.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:45.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:45.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:45.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:45.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:17:45.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:45.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:44 smithi070 ceph-mon[33250]: osdmap e49: 8 total, 8 up, 8 in 2024-05-03T17:17:45.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:44 smithi067 ceph-mon[29731]: purged_snaps scrub starts 2024-05-03T17:17:45.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:44 smithi067 ceph-mon[29731]: purged_snaps scrub ok 2024-05-03T17:17:45.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:45.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:45.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:45.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:45.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:17:45.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:45.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:45.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:17:45.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:17:45.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:44 smithi067 ceph-mon[29731]: osdmap e49: 8 total, 8 up, 8 in 2024-05-03T17:17:46.100 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:45 smithi082 ceph-mon[33535]: Detected new or changed devices on smithi070 2024-05-03T17:17:46.100 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:45 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi070 to 5696M 2024-05-03T17:17:46.100 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:45 smithi082 ceph-mon[33535]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:17:46.100 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:45 smithi082 ceph-mon[33535]: osdmap e50: 8 total, 8 up, 8 in 2024-05-03T17:17:46.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:45 smithi070 ceph-mon[33250]: Detected new or changed devices on smithi070 2024-05-03T17:17:46.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:45 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi070 to 5696M 2024-05-03T17:17:46.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:45 smithi070 ceph-mon[33250]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:17:46.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:45 smithi070 ceph-mon[33250]: osdmap e50: 8 total, 8 up, 8 in 2024-05-03T17:17:46.364 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.361+0000 7fb0830ca700 1 -- 172.21.15.82:0/1194589606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb07c1091f0 msgr2=0x7fb07c10b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:17:46.364 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.361+0000 7fb0830ca700 1 --2- 172.21.15.82:0/1194589606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb07c1091f0 0x7fb07c10b660 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb06c009990 tx=0x7fb06c009ca0 comp rx=0 tx=0).stop 2024-05-03T17:17:46.365 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.361+0000 7fb0830ca700 1 -- 172.21.15.82:0/1194589606 shutdown_connections 2024-05-03T17:17:46.365 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.361+0000 7fb0830ca700 1 --2- 172.21.15.82:0/1194589606 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb07c1091f0 0x7fb07c10b660 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:46.365 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.361+0000 7fb0830ca700 1 --2- 172.21.15.82:0/1194589606 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c103f90 0x7fb07c106380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:46.365 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.361+0000 7fb0830ca700 1 --2- 172.21.15.82:0/1194589606 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb07c1068c0 0x7fb07c108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:46.365 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.361+0000 7fb0830ca700 1 -- 172.21.15.82:0/1194589606 >> 172.21.15.82:0/1194589606 conn(0x7fb07c0fd9a0 msgr2=0x7fb07c0ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:17:46.365 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.361+0000 7fb0830ca700 1 -- 172.21.15.82:0/1194589606 shutdown_connections 2024-05-03T17:17:46.365 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.361+0000 7fb0830ca700 1 -- 172.21.15.82:0/1194589606 wait complete. 2024-05-03T17:17:46.365 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb0830ca700 1 Processor -- start 2024-05-03T17:17:46.365 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb0830ca700 1 -- start start 2024-05-03T17:17:46.365 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb0830ca700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb07c103f90 0x7fb07c1038f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:17:46.365 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb0830ca700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb07c1068c0 0x7fb07c101f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:17:46.365 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb0830ca700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c1091f0 0x7fb07c102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb0830ca700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c10efc0 con 0x7fb07c1091f0 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb0830ca700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c10ee40 con 0x7fb07c1068c0 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb0830ca700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb07c10f140 con 0x7fb07c103f90 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb080e66700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb07c103f90 0x7fb07c1038f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb080e66700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb07c103f90 0x7fb07c1038f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:45018/0 (socket says 172.21.15.82:45018) 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb080e66700 1 -- 172.21.15.82:0/664857961 learned_addr learned my addr 172.21.15.82:0/664857961 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb07bfff700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb07c1068c0 0x7fb07c101f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb081667700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c1091f0 0x7fb07c102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb080e66700 1 -- 172.21.15.82:0/664857961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb07c1068c0 msgr2=0x7fb07c101f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb080e66700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb07c1068c0 0x7fb07c101f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb080e66700 1 -- 172.21.15.82:0/664857961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c1091f0 msgr2=0x7fb07c102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb080e66700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c1091f0 0x7fb07c102480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb080e66700 1 -- 172.21.15.82:0/664857961 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07c102d30 con 0x7fb07c103f90 2024-05-03T17:17:46.366 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb07bfff700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb07c1068c0 0x7fb07c101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:17:46.367 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb081667700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c1091f0 0x7fb07c102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:17:46.367 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb080e66700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb07c103f90 0x7fb07c1038f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb0740060b0 tx=0x7fb074007f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:17:46.367 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb079ffb700 1 -- 172.21.15.82:0/664857961 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07401a070 con 0x7fb07c103f90 2024-05-03T17:17:46.367 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb079ffb700 1 -- 172.21.15.82:0/664857961 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb07400de20 con 0x7fb07c103f90 2024-05-03T17:17:46.367 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb0830ca700 1 -- 172.21.15.82:0/664857961 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb07c1a37e0 con 0x7fb07c103f90 2024-05-03T17:17:46.367 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.362+0000 7fb079ffb700 1 -- 172.21.15.82:0/664857961 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb074014720 con 0x7fb07c103f90 2024-05-03T17:17:46.367 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.363+0000 7fb0830ca700 1 -- 172.21.15.82:0/664857961 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb07c1a3cf0 con 0x7fb07c103f90 2024-05-03T17:17:46.368 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.364+0000 7fb0830ca700 1 -- 172.21.15.82:0/664857961 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb07c06a190 con 0x7fb07c103f90 2024-05-03T17:17:46.368 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.365+0000 7fb079ffb700 1 -- 172.21.15.82:0/664857961 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fb07401d070 con 0x7fb07c103f90 2024-05-03T17:17:46.368 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.365+0000 7fb079ffb700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb0640b2880 0x7fb0640b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:17:46.368 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.366+0000 7fb079ffb700 1 -- 172.21.15.82:0/664857961 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5397+0+0 (secure 0 0 0) 0x7fb0740e1490 con 0x7fb07c103f90 2024-05-03T17:17:46.368 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.366+0000 7fb07bfff700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb0640b2880 0x7fb0640b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:17:46.368 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.366+0000 7fb07bfff700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb0640b2880 0x7fb0640b4d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb07000bcf0 tx=0x7fb07000b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:17:46.371 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.369+0000 7fb079ffb700 1 -- 172.21.15.82:0/664857961 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb074010070 con 0x7fb07c103f90 2024-05-03T17:17:46.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:45 smithi067 ceph-mon[29731]: Detected new or changed devices on smithi070 2024-05-03T17:17:46.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:45 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi070 to 5696M 2024-05-03T17:17:46.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:45 smithi067 ceph-mon[29731]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:17:46.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:45 smithi067 ceph-mon[29731]: osdmap e50: 8 total, 8 up, 8 in 2024-05-03T17:17:46.580 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:17:46.578+0000 7fb0830ca700 1 -- 172.21.15.82:0/664857961 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fb07c0638e0 con 0x7fb0640b2880 2024-05-03T17:17:47.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:17:47.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:17:47.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:47.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:17:47.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:17:47.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:47.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:17:47.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:17:47.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:48.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:47 smithi082 ceph-mon[33535]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:17:48.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:47 smithi082 ceph-mon[33535]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:17:48.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:47 smithi070 ceph-mon[33250]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:17:48.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:47 smithi070 ceph-mon[33250]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:17:48.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:47 smithi067 ceph-mon[29731]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:17:48.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:47 smithi067 ceph-mon[29731]: from='client.24283 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:17:50.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:49 smithi082 ceph-mon[33535]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:17:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:49 smithi082 ceph-mon[33535]: from='client.? 172.21.15.82:0/206200227' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9d4ce56-e940-48d1-a9b5-437139e3fed0"}]: dispatch 2024-05-03T17:17:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:49 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9d4ce56-e940-48d1-a9b5-437139e3fed0"}]: dispatch 2024-05-03T17:17:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:49 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9d4ce56-e940-48d1-a9b5-437139e3fed0"}]': finished 2024-05-03T17:17:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:49 smithi082 ceph-mon[33535]: osdmap e51: 9 total, 8 up, 9 in 2024-05-03T17:17:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:17:50.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:49 smithi070 ceph-mon[33250]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:17:50.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:49 smithi070 ceph-mon[33250]: from='client.? 172.21.15.82:0/206200227' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9d4ce56-e940-48d1-a9b5-437139e3fed0"}]: dispatch 2024-05-03T17:17:50.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:49 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9d4ce56-e940-48d1-a9b5-437139e3fed0"}]: dispatch 2024-05-03T17:17:50.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:49 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9d4ce56-e940-48d1-a9b5-437139e3fed0"}]': finished 2024-05-03T17:17:50.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:49 smithi070 ceph-mon[33250]: osdmap e51: 9 total, 8 up, 9 in 2024-05-03T17:17:50.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:17:50.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:49 smithi067 ceph-mon[29731]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:17:50.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:49 smithi067 ceph-mon[29731]: from='client.? 172.21.15.82:0/206200227' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9d4ce56-e940-48d1-a9b5-437139e3fed0"}]: dispatch 2024-05-03T17:17:50.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:49 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9d4ce56-e940-48d1-a9b5-437139e3fed0"}]: dispatch 2024-05-03T17:17:50.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:49 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9d4ce56-e940-48d1-a9b5-437139e3fed0"}]': finished 2024-05-03T17:17:50.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:49 smithi067 ceph-mon[29731]: osdmap e51: 9 total, 8 up, 9 in 2024-05-03T17:17:50.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:17:51.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:50 smithi082 ceph-mon[33535]: from='client.? 172.21.15.82:0/3987719565' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:17:51.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:50 smithi070 ceph-mon[33250]: from='client.? 172.21.15.82:0/3987719565' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:17:51.354 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:50 smithi067 ceph-mon[29731]: from='client.? 172.21.15.82:0/3987719565' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:17:52.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:51 smithi082 ceph-mon[33535]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T17:17:52.307 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:51 smithi070 ceph-mon[33250]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T17:17:52.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:51 smithi067 ceph-mon[29731]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T17:17:54.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:53 smithi082 ceph-mon[33535]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-05-03T17:17:54.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:53 smithi070 ceph-mon[33250]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-05-03T17:17:54.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:53 smithi067 ceph-mon[29731]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-05-03T17:17:56.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:55 smithi082 ceph-mon[33535]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-03T17:17:56.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:55 smithi070 ceph-mon[33250]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-03T17:17:56.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:55 smithi067 ceph-mon[29731]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-03T17:17:57.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:57 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-03T17:17:57.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:57 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:57.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:57 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-03T17:17:57.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:57 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:57.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:57 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-03T17:17:57.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:57 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:17:58.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:58 smithi082 ceph-mon[33535]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:17:58.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:17:58 smithi082 ceph-mon[33535]: Deploying daemon osd.8 on smithi082 2024-05-03T17:17:58.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:58 smithi070 ceph-mon[33250]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:17:58.310 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:17:58 smithi070 ceph-mon[33250]: Deploying daemon osd.8 on smithi082 2024-05-03T17:17:58.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:58 smithi067 ceph-mon[29731]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:17:58.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:17:58 smithi067 ceph-mon[29731]: Deploying daemon osd.8 on smithi082 2024-05-03T17:18:00.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:00 smithi082 ceph-mon[33535]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:18:00.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:00 smithi070 ceph-mon[33250]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:18:00.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:00 smithi067 ceph-mon[29731]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:18:02.102 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:02 smithi082 ceph-mon[33535]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:02.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:02 smithi070 ceph-mon[33250]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:02.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:02 smithi067 ceph-mon[29731]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:04.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:03 smithi082 ceph-mon[33535]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:04.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:03 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:04.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:03 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:04.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:03 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:04.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:03 smithi070 ceph-mon[33250]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:04.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:03 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:04.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:03 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:04.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:03 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:04.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:03 smithi067 ceph-mon[29731]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:04.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:03 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:04.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:03 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:04.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:03 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:06.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:05 smithi082 ceph-mon[33535]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:06.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:05 smithi070 ceph-mon[33250]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:06.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:05 smithi067 ceph-mon[29731]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:06.851 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.850+0000 7fb079ffb700 1 -- 172.21.15.82:0/664857961 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb07c0638e0 con 0x7fb0640b2880 2024-05-03T17:18:06.852 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 8 on host 'smithi082' 2024-05-03T17:18:06.853 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.853+0000 7fb0830ca700 1 -- 172.21.15.82:0/664857961 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb0640b2880 msgr2=0x7fb0640b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:18:06.853 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.853+0000 7fb0830ca700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb0640b2880 0x7fb0640b4d40 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb07000bcf0 tx=0x7fb07000b2c0 comp rx=0 tx=0).stop 2024-05-03T17:18:06.853 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.853+0000 7fb0830ca700 1 -- 172.21.15.82:0/664857961 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb07c103f90 msgr2=0x7fb07c1038f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:18:06.853 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.853+0000 7fb0830ca700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb07c103f90 0x7fb07c1038f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb0740060b0 tx=0x7fb074007f90 comp rx=0 tx=0).stop 2024-05-03T17:18:06.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.853+0000 7fb0830ca700 1 -- 172.21.15.82:0/664857961 shutdown_connections 2024-05-03T17:18:06.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.853+0000 7fb0830ca700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb0640b2880 0x7fb0640b4d40 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:06.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.853+0000 7fb0830ca700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb07c1091f0 0x7fb07c102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:06.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.853+0000 7fb0830ca700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb07c103f90 0x7fb07c1038f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:06.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.853+0000 7fb0830ca700 1 --2- 172.21.15.82:0/664857961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb07c1068c0 0x7fb07c101f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:06.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.853+0000 7fb0830ca700 1 -- 172.21.15.82:0/664857961 >> 172.21.15.82:0/664857961 conn(0x7fb07c0fd9a0 msgr2=0x7fb07c1049f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:18:06.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.853+0000 7fb0830ca700 1 -- 172.21.15.82:0/664857961 shutdown_connections 2024-05-03T17:18:06.854 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:18:06.853+0000 7fb0830ca700 1 -- 172.21.15.82:0/664857961 wait complete. 2024-05-03T17:18:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:18:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:06 smithi070 ceph-mon[33250]: from='osd.8 [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-03T17:18:07.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:06 smithi070 ceph-mon[33250]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-03T17:18:07.178 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:07.178 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:07.178 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:07.178 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:18:07.179 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:07.179 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:06 smithi082 ceph-mon[33535]: from='osd.8 [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-03T17:18:07.179 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:06 smithi082 ceph-mon[33535]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-03T17:18:07.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:07.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:07.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:07.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:18:07.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:07.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:06 smithi067 ceph-mon[29731]: from='osd.8 [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-03T17:18:07.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:06 smithi067 ceph-mon[29731]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-03T17:18:07.521 DEBUG:teuthology.orchestra.run.smithi082:osd.8> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.8.service 2024-05-03T17:18:07.523 INFO:tasks.cephadm:Deploying osd.9 on smithi067 with /dev/vg_nvme/lv_1... 2024-05-03T17:18:07.523 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-03T17:18:07.779 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:18:08.021 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:07 smithi082 ceph-mon[33535]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:08.021 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:08.021 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:08.021 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:07 smithi082 ceph-mon[33535]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-03T17:18:08.021 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:07 smithi082 ceph-mon[33535]: from='osd.8 [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:18:08.021 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:07 smithi082 ceph-mon[33535]: osdmap e52: 9 total, 8 up, 9 in 2024-05-03T17:18:08.021 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:18:08.021 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:07 smithi082 ceph-mon[33535]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:18:08.021 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:08.021 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:18:07 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:18:07.829+0000 7fcc40278700 -1 osd.8 0 waiting for initial osdmap 2024-05-03T17:18:08.022 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:18:07 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:18:07.839+0000 7fcc3c06b700 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T17:18:08.072 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:07 smithi067 ceph-mon[29731]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:08.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:08.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:08.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:07 smithi067 ceph-mon[29731]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-03T17:18:08.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:07 smithi067 ceph-mon[29731]: from='osd.8 [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:18:08.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:07 smithi067 ceph-mon[29731]: osdmap e52: 9 total, 8 up, 9 in 2024-05-03T17:18:08.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:18:08.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:07 smithi067 ceph-mon[29731]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:18:08.073 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:07 smithi070 ceph-mon[33250]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:07 smithi070 ceph-mon[33250]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-03T17:18:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:07 smithi070 ceph-mon[33250]: from='osd.8 [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:18:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:07 smithi070 ceph-mon[33250]: osdmap e52: 9 total, 8 up, 9 in 2024-05-03T17:18:08.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:18:08.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:07 smithi070 ceph-mon[33250]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:18:08.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:09.125 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:08 smithi067 ceph-mon[29731]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T17:18:09.125 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:08 smithi067 ceph-mon[29731]: osdmap e53: 9 total, 8 up, 9 in 2024-05-03T17:18:09.125 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:08 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:18:09.125 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:08 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:18:09.142 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:08 smithi082 ceph-mon[33535]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T17:18:09.143 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:08 smithi082 ceph-mon[33535]: osdmap e53: 9 total, 8 up, 9 in 2024-05-03T17:18:09.143 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:08 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:18:09.143 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:08 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:18:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:08 smithi070 ceph-mon[33250]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T17:18:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:08 smithi070 ceph-mon[33250]: osdmap e53: 9 total, 8 up, 9 in 2024-05-03T17:18:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:08 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:18:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:08 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:18:10.048 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-03T17:18:10.049 INFO:teuthology.orchestra.run.smithi067.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-03T17:18:10.049 INFO:teuthology.orchestra.run.smithi067.stderr: stderr: 10+0 records in 2024-05-03T17:18:10.049 INFO:teuthology.orchestra.run.smithi067.stderr:10+0 records out 2024-05-03T17:18:10.049 INFO:teuthology.orchestra.run.smithi067.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0120996 s, 867 MB/s 2024-05-03T17:18:10.049 INFO:teuthology.orchestra.run.smithi067.stderr:--> Zapping successful for: 2024-05-03T17:18:10.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:09 smithi067 ceph-mon[29731]: purged_snaps scrub starts 2024-05-03T17:18:10.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:09 smithi067 ceph-mon[29731]: purged_snaps scrub ok 2024-05-03T17:18:10.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:09 smithi067 ceph-mon[29731]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:10.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:09 smithi067 ceph-mon[29731]: osd.8 [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286] boot 2024-05-03T17:18:10.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:09 smithi067 ceph-mon[29731]: osdmap e54: 9 total, 9 up, 9 in 2024-05-03T17:18:10.099 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:18:10.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:09 smithi082 ceph-mon[33535]: purged_snaps scrub starts 2024-05-03T17:18:10.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:09 smithi082 ceph-mon[33535]: purged_snaps scrub ok 2024-05-03T17:18:10.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:09 smithi082 ceph-mon[33535]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:10.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:09 smithi082 ceph-mon[33535]: osd.8 [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286] boot 2024-05-03T17:18:10.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:09 smithi082 ceph-mon[33535]: osdmap e54: 9 total, 9 up, 9 in 2024-05-03T17:18:10.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:18:10.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:09 smithi070 ceph-mon[33250]: purged_snaps scrub starts 2024-05-03T17:18:10.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:09 smithi070 ceph-mon[33250]: purged_snaps scrub ok 2024-05-03T17:18:10.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:09 smithi070 ceph-mon[33250]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-03T17:18:10.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:09 smithi070 ceph-mon[33250]: osd.8 [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286] boot 2024-05-03T17:18:10.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:09 smithi070 ceph-mon[33250]: osdmap e54: 9 total, 9 up, 9 in 2024-05-03T17:18:10.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-03T17:18:10.939 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi067:vg_nvme/lv_1 2024-05-03T17:18:11.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:10 smithi082 ceph-mon[33535]: osdmap e55: 9 total, 9 up, 9 in 2024-05-03T17:18:11.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:11.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:11.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:11.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:11.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:11.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:11.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:11.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:18:11.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:10 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:11.197 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:18:11.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:10 smithi067 ceph-mon[29731]: osdmap e55: 9 total, 9 up, 9 in 2024-05-03T17:18:11.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:11.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:11.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:11.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:11.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:11.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:11.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:11.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:18:11.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:10 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:10 smithi070 ceph-mon[33250]: osdmap e55: 9 total, 9 up, 9 in 2024-05-03T17:18:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:11.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:18:11.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:10 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:11.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:11 smithi067 ceph-mon[29731]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 254 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:11.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:11 smithi067 ceph-mon[29731]: Detected new or changed devices on smithi082 2024-05-03T17:18:11.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:11 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi082 to 5696M 2024-05-03T17:18:11.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:11 smithi067 ceph-mon[29731]: osdmap e56: 9 total, 9 up, 9 in 2024-05-03T17:18:12.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:11 smithi082 ceph-mon[33535]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 254 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:12.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:11 smithi082 ceph-mon[33535]: Detected new or changed devices on smithi082 2024-05-03T17:18:12.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:11 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi082 to 5696M 2024-05-03T17:18:12.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:11 smithi082 ceph-mon[33535]: osdmap e56: 9 total, 9 up, 9 in 2024-05-03T17:18:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:11 smithi070 ceph-mon[33250]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 254 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:11 smithi070 ceph-mon[33250]: Detected new or changed devices on smithi082 2024-05-03T17:18:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:11 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi082 to 5696M 2024-05-03T17:18:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:11 smithi070 ceph-mon[33250]: osdmap e56: 9 total, 9 up, 9 in 2024-05-03T17:18:13.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.135+0000 7fcc39535700 1 -- 172.21.15.67:0/1956417564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc34104010 msgr2=0x7fcc34106400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:18:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.135+0000 7fcc39535700 1 --2- 172.21.15.67:0/1956417564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc34104010 0x7fcc34106400 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcc24009990 tx=0x7fcc24009ca0 comp rx=0 tx=0).stop 2024-05-03T17:18:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.135+0000 7fcc39535700 1 -- 172.21.15.67:0/1956417564 shutdown_connections 2024-05-03T17:18:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.135+0000 7fcc39535700 1 --2- 172.21.15.67:0/1956417564 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc34109270 0x7fcc3410b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.135+0000 7fcc39535700 1 --2- 172.21.15.67:0/1956417564 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc34104010 0x7fcc34106400 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.135+0000 7fcc39535700 1 --2- 172.21.15.67:0/1956417564 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc34106940 0x7fcc34108d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.135+0000 7fcc39535700 1 -- 172.21.15.67:0/1956417564 >> 172.21.15.67:0/1956417564 conn(0x7fcc340fd9a0 msgr2=0x7fcc340ffde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:18:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.135+0000 7fcc39535700 1 -- 172.21.15.67:0/1956417564 shutdown_connections 2024-05-03T17:18:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.135+0000 7fcc39535700 1 -- 172.21.15.67:0/1956417564 wait complete. 2024-05-03T17:18:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.136+0000 7fcc39535700 1 Processor -- start 2024-05-03T17:18:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.136+0000 7fcc39535700 1 -- start start 2024-05-03T17:18:13.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.136+0000 7fcc39535700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc34104010 0x7fcc34103970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:18:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.136+0000 7fcc39535700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc34106940 0x7fcc34101fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:18:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.136+0000 7fcc39535700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc34109270 0x7fcc34102500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:18:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.136+0000 7fcc39535700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3410f000 con 0x7fcc34104010 2024-05-03T17:18:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.136+0000 7fcc39535700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3410ee80 con 0x7fcc34106940 2024-05-03T17:18:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.136+0000 7fcc39535700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcc3410f180 con 0x7fcc34109270 2024-05-03T17:18:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.136+0000 7fcc32ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc34104010 0x7fcc34103970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:18:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.136+0000 7fcc32ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc34104010 0x7fcc34103970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47930/0 (socket says 172.21.15.67:47930) 2024-05-03T17:18:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.136+0000 7fcc32ffd700 1 -- 172.21.15.67:0/2610840893 learned_addr learned my addr 172.21.15.67:0/2610840893 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:18:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.136+0000 7fcc327fc700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc34106940 0x7fcc34101fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:18:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc337fe700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc34109270 0x7fcc34102500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:18:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc32ffd700 1 -- 172.21.15.67:0/2610840893 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc34109270 msgr2=0x7fcc34102500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:18:13.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc32ffd700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc34109270 0x7fcc34102500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc32ffd700 1 -- 172.21.15.67:0/2610840893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc34106940 msgr2=0x7fcc34101fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:18:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc32ffd700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc34106940 0x7fcc34101fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc32ffd700 1 -- 172.21.15.67:0/2610840893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc34102db0 con 0x7fcc34104010 2024-05-03T17:18:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc327fc700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc34106940 0x7fcc34101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:18:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc337fe700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc34109270 0x7fcc34102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:18:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc32ffd700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc34104010 0x7fcc34103970 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400b760 tx=0x7fcc24005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:18:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc1bfff700 1 -- 172.21.15.67:0/2610840893 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24026070 con 0x7fcc34104010 2024-05-03T17:18:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc1bfff700 1 -- 172.21.15.67:0/2610840893 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc2400fc40 con 0x7fcc34104010 2024-05-03T17:18:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc39535700 1 -- 172.21.15.67:0/2610840893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc341a3810 con 0x7fcc34104010 2024-05-03T17:18:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc1bfff700 1 -- 172.21.15.67:0/2610840893 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc24017650 con 0x7fcc34104010 2024-05-03T17:18:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.137+0000 7fcc39535700 1 -- 172.21.15.67:0/2610840893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc341a3ce0 con 0x7fcc34104010 2024-05-03T17:18:13.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.138+0000 7fcc39535700 1 -- 172.21.15.67:0/2610840893 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc3406a190 con 0x7fcc34104010 2024-05-03T17:18:13.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.140+0000 7fcc1bfff700 1 -- 172.21.15.67:0/2610840893 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fcc24029070 con 0x7fcc34104010 2024-05-03T17:18:13.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.140+0000 7fcc1bfff700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fcc1c0b2880 0x7fcc1c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:18:13.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.140+0000 7fcc327fc700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fcc1c0b2880 0x7fcc1c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:18:13.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.140+0000 7fcc1bfff700 1 -- 172.21.15.67:0/2610840893 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5829+0+0 (secure 0 0 0) 0x7fcc240e2a70 con 0x7fcc34104010 2024-05-03T17:18:13.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.141+0000 7fcc327fc700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fcc1c0b2880 0x7fcc1c0b4d40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009cb0 tx=0x7fcc28009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:18:13.145 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.144+0000 7fcc1bfff700 1 -- 172.21.15.67:0/2610840893 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fcc24013070 con 0x7fcc34104010 2024-05-03T17:18:13.369 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:13.368+0000 7fcc39535700 1 -- 172.21.15.67:0/2610840893 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fcc340638e0 con 0x7fcc1c0b2880 2024-05-03T17:18:14.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:13 smithi067 ceph-mon[29731]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 254 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:14.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:18:14.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:18:14.173 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:14.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:13 smithi082 ceph-mon[33535]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 254 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:14.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:18:14.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:18:14.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:14.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:13 smithi070 ceph-mon[33250]: pgmap v199: 1 pgs: 1 remapped+peering; 577 KiB data, 254 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:14.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:18:14.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:18:14.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:15.157 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:14 smithi067 ceph-mon[29731]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:18:15.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:14 smithi082 ceph-mon[33535]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:18:15.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:14 smithi070 ceph-mon[33250]: from='client.14403 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi067:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:18:16.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:15 smithi067 ceph-mon[29731]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T17:18:16.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:15 smithi082 ceph-mon[33535]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T17:18:16.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:15 smithi070 ceph-mon[33250]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-03T17:18:17.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:16 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/918945417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09596d19-f2d7-485e-a028-9ce5468c6651"}]: dispatch 2024-05-03T17:18:17.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:16 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/918945417' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09596d19-f2d7-485e-a028-9ce5468c6651"}]': finished 2024-05-03T17:18:17.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:16 smithi067 ceph-mon[29731]: osdmap e57: 10 total, 9 up, 10 in 2024-05-03T17:18:17.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:16 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:17.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:16 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/918945417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09596d19-f2d7-485e-a028-9ce5468c6651"}]: dispatch 2024-05-03T17:18:17.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:16 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/918945417' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09596d19-f2d7-485e-a028-9ce5468c6651"}]': finished 2024-05-03T17:18:17.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:16 smithi082 ceph-mon[33535]: osdmap e57: 10 total, 9 up, 10 in 2024-05-03T17:18:17.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:16 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:16 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/918945417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "09596d19-f2d7-485e-a028-9ce5468c6651"}]: dispatch 2024-05-03T17:18:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:16 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/918945417' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "09596d19-f2d7-485e-a028-9ce5468c6651"}]': finished 2024-05-03T17:18:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:16 smithi070 ceph-mon[33250]: osdmap e57: 10 total, 9 up, 10 in 2024-05-03T17:18:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:16 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:18.166 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:17 smithi082 ceph-mon[33535]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-05-03T17:18:18.166 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:17 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1111487539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:18:18.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:17 smithi067 ceph-mon[29731]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-05-03T17:18:18.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:17 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1111487539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:18:18.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:17 smithi070 ceph-mon[33250]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-05-03T17:18:18.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:17 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1111487539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:18:20.110 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:19 smithi070 ceph-mon[33250]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-03T17:18:20.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:19 smithi067 ceph-mon[29731]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-03T17:18:20.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:19 smithi082 ceph-mon[33535]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-03T17:18:22.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:21 smithi067 ceph-mon[29731]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-03T17:18:22.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:21 smithi082 ceph-mon[33535]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-03T17:18:22.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:21 smithi070 ceph-mon[33250]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-05-03T17:18:24.134 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:23 smithi067 ceph-mon[29731]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:18:24.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:23 smithi082 ceph-mon[33535]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:18:24.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:23 smithi070 ceph-mon[33250]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:18:25.140 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-03T17:18:25.141 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:25.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-03T17:18:25.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:25.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-03T17:18:25.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:26.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:25 smithi067 ceph-mon[29731]: Deploying daemon osd.9 on smithi067 2024-05-03T17:18:26.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:25 smithi067 ceph-mon[29731]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:26.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:25 smithi082 ceph-mon[33535]: Deploying daemon osd.9 on smithi067 2024-05-03T17:18:26.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:25 smithi082 ceph-mon[33535]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:26.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:25 smithi070 ceph-mon[33250]: Deploying daemon osd.9 on smithi067 2024-05-03T17:18:26.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:25 smithi070 ceph-mon[33250]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:28.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:27 smithi067 ceph-mon[29731]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:28.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:27 smithi082 ceph-mon[33535]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:28.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:27 smithi070 ceph-mon[33250]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:30.103 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:29 smithi067 ceph-mon[29731]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:30.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:29 smithi082 ceph-mon[33535]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:30.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:29 smithi070 ceph-mon[33250]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:31.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:31 smithi082 ceph-mon[33535]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:31.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:31 smithi070 ceph-mon[33250]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:31.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:31.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:31.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:31 smithi067 ceph-mon[29731]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:31.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:33.619 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:33 smithi067 ceph-mon[29731]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:33.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:33 smithi082 ceph-mon[33535]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:33.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:33 smithi070 ceph-mon[33250]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:34.793 INFO:teuthology.orchestra.run.smithi067.stdout:Created osd(s) 9 on host 'smithi067' 2024-05-03T17:18:34.793 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.792+0000 7fcc1bfff700 1 -- 172.21.15.67:0/2610840893 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcc340638e0 con 0x7fcc1c0b2880 2024-05-03T17:18:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.795+0000 7fcc39535700 1 -- 172.21.15.67:0/2610840893 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fcc1c0b2880 msgr2=0x7fcc1c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:18:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.795+0000 7fcc39535700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fcc1c0b2880 0x7fcc1c0b4d40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009cb0 tx=0x7fcc28009260 comp rx=0 tx=0).stop 2024-05-03T17:18:34.795 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.795+0000 7fcc39535700 1 -- 172.21.15.67:0/2610840893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc34104010 msgr2=0x7fcc34103970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:18:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.795+0000 7fcc39535700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc34104010 0x7fcc34103970 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcc2400b760 tx=0x7fcc24005e80 comp rx=0 tx=0).stop 2024-05-03T17:18:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.795+0000 7fcc39535700 1 -- 172.21.15.67:0/2610840893 shutdown_connections 2024-05-03T17:18:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.795+0000 7fcc39535700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fcc1c0b2880 0x7fcc1c0b4d40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.795+0000 7fcc39535700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcc34109270 0x7fcc34102500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.795+0000 7fcc39535700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcc34104010 0x7fcc34103970 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.795+0000 7fcc39535700 1 --2- 172.21.15.67:0/2610840893 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcc34106940 0x7fcc34101fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.795+0000 7fcc39535700 1 -- 172.21.15.67:0/2610840893 >> 172.21.15.67:0/2610840893 conn(0x7fcc340fd9a0 msgr2=0x7fcc34104950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:18:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.795+0000 7fcc39535700 1 -- 172.21.15.67:0/2610840893 shutdown_connections 2024-05-03T17:18:34.796 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:18:34.795+0000 7fcc39535700 1 -- 172.21.15.67:0/2610840893 wait complete. 2024-05-03T17:18:35.150 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.150 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.150 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:35.150 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:18:35.150 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.150 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:34 smithi067 ceph-mon[29731]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-03T17:18:35.150 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.150 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:35.150 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:35.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:18:35.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:34 smithi082 ceph-mon[33535]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-03T17:18:35.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:35.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:35.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:18:35.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:34 smithi070 ceph-mon[33250]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-03T17:18:35.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:35.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:35.510 DEBUG:teuthology.orchestra.run.smithi067:osd.9> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.9.service 2024-05-03T17:18:35.512 INFO:tasks.cephadm:Deploying osd.10 on smithi070 with /dev/vg_nvme/lv_1... 2024-05-03T17:18:35.512 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-03T17:18:35.758 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T17:18:36.037 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:35 smithi067 ceph-mon[29731]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:36.037 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:35 smithi067 ceph-mon[29731]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-03T17:18:36.037 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:35 smithi067 ceph-mon[29731]: osdmap e58: 10 total, 9 up, 10 in 2024-05-03T17:18:36.037 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:35 smithi067 ceph-mon[29731]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:18:36.037 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:35 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:36.143 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:35 smithi070 ceph-mon[33250]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:36.143 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:35 smithi070 ceph-mon[33250]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-03T17:18:36.143 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:35 smithi070 ceph-mon[33250]: osdmap e58: 10 total, 9 up, 10 in 2024-05-03T17:18:36.144 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:35 smithi070 ceph-mon[33250]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:18:36.144 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:35 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:36.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:35 smithi082 ceph-mon[33535]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:36.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:35 smithi082 ceph-mon[33535]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-03T17:18:36.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:35 smithi082 ceph-mon[33535]: osdmap e58: 10 total, 9 up, 10 in 2024-05-03T17:18:36.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:35 smithi082 ceph-mon[33535]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]: dispatch 2024-05-03T17:18:36.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:35 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:36.453 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:18:36 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:18:36.269+0000 7f3e32073700 -1 osd.9 0 waiting for initial osdmap 2024-05-03T17:18:36.453 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:18:36 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:18:36.279+0000 7f3e2965d700 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T17:18:36.998 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:36 smithi070 ceph-mon[33250]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T17:18:36.998 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:36 smithi070 ceph-mon[33250]: osdmap e59: 10 total, 9 up, 10 in 2024-05-03T17:18:36.998 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:36.999 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:36.999 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:36 smithi070 ceph-mon[33250]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' 2024-05-03T17:18:37.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:36 smithi067 ceph-mon[29731]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T17:18:37.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:36 smithi067 ceph-mon[29731]: osdmap e59: 10 total, 9 up, 10 in 2024-05-03T17:18:37.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:37.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:37.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:36 smithi067 ceph-mon[29731]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' 2024-05-03T17:18:37.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:36 smithi082 ceph-mon[33535]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi067", "root=default"]}]': finished 2024-05-03T17:18:37.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:36 smithi082 ceph-mon[33535]: osdmap e59: 10 total, 9 up, 10 in 2024-05-03T17:18:37.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:37.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:37.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:36 smithi082 ceph-mon[33535]: from='osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198]' entity='osd.9' 2024-05-03T17:18:37.859 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-03T17:18:37.859 INFO:teuthology.orchestra.run.smithi070.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-03T17:18:37.859 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10+0 records in 2024-05-03T17:18:37.859 INFO:teuthology.orchestra.run.smithi070.stderr:10+0 records out 2024-05-03T17:18:37.859 INFO:teuthology.orchestra.run.smithi070.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00859179 s, 1.2 GB/s 2024-05-03T17:18:37.859 INFO:teuthology.orchestra.run.smithi070.stderr:--> Zapping successful for: 2024-05-03T17:18:38.196 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:37 smithi070 ceph-mon[33250]: purged_snaps scrub starts 2024-05-03T17:18:38.197 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:37 smithi070 ceph-mon[33250]: purged_snaps scrub ok 2024-05-03T17:18:38.197 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:37 smithi070 ceph-mon[33250]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:38.197 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:37 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:38.197 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:37 smithi070 ceph-mon[33250]: osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198] boot 2024-05-03T17:18:38.197 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:37 smithi070 ceph-mon[33250]: osdmap e60: 10 total, 10 up, 10 in 2024-05-03T17:18:38.197 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:37 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:38.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:37 smithi067 ceph-mon[29731]: purged_snaps scrub starts 2024-05-03T17:18:38.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:37 smithi067 ceph-mon[29731]: purged_snaps scrub ok 2024-05-03T17:18:38.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:37 smithi067 ceph-mon[29731]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:38.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:37 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:38.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:37 smithi067 ceph-mon[29731]: osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198] boot 2024-05-03T17:18:38.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:37 smithi067 ceph-mon[29731]: osdmap e60: 10 total, 10 up, 10 in 2024-05-03T17:18:38.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:37 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:38.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:37 smithi082 ceph-mon[33535]: purged_snaps scrub starts 2024-05-03T17:18:38.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:37 smithi082 ceph-mon[33535]: purged_snaps scrub ok 2024-05-03T17:18:38.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:37 smithi082 ceph-mon[33535]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-03T17:18:38.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:37 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:38.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:37 smithi082 ceph-mon[33535]: osd.9 [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198] boot 2024-05-03T17:18:38.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:37 smithi082 ceph-mon[33535]: osdmap e60: 10 total, 10 up, 10 in 2024-05-03T17:18:38.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:37 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-03T17:18:38.516 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi070:vg_nvme/lv_1 2024-05-03T17:18:38.750 INFO:teuthology.orchestra.run.smithi070.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.b/config 2024-05-03T17:18:39.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: osdmap e61: 10 total, 10 up, 10 in 2024-05-03T17:18:39.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:39.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: Detected new or changed devices on smithi067 2024-05-03T17:18:39.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:39.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:39.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:39.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:39.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi067 to 4272M 2024-05-03T17:18:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:18:39.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:39.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: osdmap e61: 10 total, 10 up, 10 in 2024-05-03T17:18:39.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: Detected new or changed devices on smithi067 2024-05-03T17:18:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi067 to 4272M 2024-05-03T17:18:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:18:39.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:39.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: osdmap e61: 10 total, 10 up, 10 in 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: Detected new or changed devices on smithi067 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi067 to 4272M 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:18:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:40.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.409+0000 7fa4a82b3700 1 -- 172.21.15.70:0/443753783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4a01050a0 msgr2=0x7fa4a010b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:18:40.412 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.409+0000 7fa4a82b3700 1 --2- 172.21.15.70:0/443753783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4a01050a0 0x7fa4a010b950 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa49800b0d0 tx=0x7fa49800b3e0 comp rx=0 tx=0).stop 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.410+0000 7fa4a82b3700 1 -- 172.21.15.70:0/443753783 shutdown_connections 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.410+0000 7fa4a82b3700 1 --2- 172.21.15.70:0/443753783 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4a01050a0 0x7fa4a010b950 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.410+0000 7fa4a82b3700 1 --2- 172.21.15.70:0/443753783 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4a0103500 0x7fa4a0103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.410+0000 7fa4a82b3700 1 --2- 172.21.15.70:0/443753783 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4a0104700 0x7fa4a0104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.410+0000 7fa4a82b3700 1 -- 172.21.15.70:0/443753783 >> 172.21.15.70:0/443753783 conn(0x7fa4a00feaa0 msgr2=0x7fa4a0100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.410+0000 7fa4a82b3700 1 -- 172.21.15.70:0/443753783 shutdown_connections 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.410+0000 7fa4a82b3700 1 -- 172.21.15.70:0/443753783 wait complete. 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.410+0000 7fa4a82b3700 1 Processor -- start 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.410+0000 7fa4a82b3700 1 -- start start 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a82b3700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4a0103500 0x7fa4a0199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a82b3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4a0104700 0x7fa4a0199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a82b3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4a01050a0 0x7fa4a01a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:18:40.413 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a82b3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a010f090 con 0x7fa4a0104700 2024-05-03T17:18:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a82b3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a010ef10 con 0x7fa4a01050a0 2024-05-03T17:18:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a82b3700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a010f210 con 0x7fa4a0103500 2024-05-03T17:18:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a6850700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4a01050a0 0x7fa4a01a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:18:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a6850700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4a01050a0 0x7fa4a01a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.70:33364/0 (socket says 172.21.15.70:33364) 2024-05-03T17:18:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a6850700 1 -- 172.21.15.70:0/1289575419 learned_addr learned my addr 172.21.15.70:0/1289575419 (peer_addr_for_me v2:172.21.15.70:0/0) 2024-05-03T17:18:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a584e700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4a0104700 0x7fa4a0199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:18:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a604f700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4a0103500 0x7fa4a0199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:18:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a6850700 1 -- 172.21.15.70:0/1289575419 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4a0103500 msgr2=0x7fa4a0199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:18:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a6850700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4a0103500 0x7fa4a0199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a6850700 1 -- 172.21.15.70:0/1289575419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4a0104700 msgr2=0x7fa4a0199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:18:40.414 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a6850700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4a0104700 0x7fa4a0199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:40.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a6850700 1 -- 172.21.15.70:0/1289575419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a01a13c0 con 0x7fa4a01050a0 2024-05-03T17:18:40.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a604f700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4a0103500 0x7fa4a0199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:18:40.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a6850700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4a01050a0 0x7fa4a01a0d20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa498009fd0 tx=0x7fa498004e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:18:40.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4977fe700 1 -- 172.21.15.70:0/1289575419 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49801f030 con 0x7fa4a01050a0 2024-05-03T17:18:40.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4977fe700 1 -- 172.21.15.70:0/1289575419 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa498007a80 con 0x7fa4a01050a0 2024-05-03T17:18:40.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4a82b3700 1 -- 172.21.15.70:0/1289575419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4a01a1650 con 0x7fa4a01050a0 2024-05-03T17:18:40.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.411+0000 7fa4977fe700 1 -- 172.21.15.70:0/1289575419 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498004810 con 0x7fa4a01050a0 2024-05-03T17:18:40.415 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.412+0000 7fa4a82b3700 1 -- 172.21.15.70:0/1289575419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4a01a1b80 con 0x7fa4a01050a0 2024-05-03T17:18:40.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.412+0000 7fa4a82b3700 1 -- 172.21.15.70:0/1289575419 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4a006a190 con 0x7fa4a01050a0 2024-05-03T17:18:40.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.414+0000 7fa4977fe700 1 -- 172.21.15.70:0/1289575419 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa498003a00 con 0x7fa4a01050a0 2024-05-03T17:18:40.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.415+0000 7fa4977fe700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa48c0b2900 0x7fa48c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:18:40.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.415+0000 7fa4977fe700 1 -- 172.21.15.70:0/1289575419 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6261+0+0 (secure 0 0 0) 0x7fa49800f020 con 0x7fa4a01050a0 2024-05-03T17:18:40.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.415+0000 7fa4a604f700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa48c0b2900 0x7fa48c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:18:40.416 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.416+0000 7fa4a604f700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa48c0b2900 0x7fa48c0b4dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0104560 tx=0x7fa49c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:18:40.419 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.419+0000 7fa4977fe700 1 -- 172.21.15.70:0/1289575419 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa498029020 con 0x7fa4a01050a0 2024-05-03T17:18:40.633 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:40.632+0000 7fa4a82b3700 1 -- 172.21.15.70:0/1289575419 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fa4a00639a0 con 0x7fa48c0b2900 2024-05-03T17:18:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:41 smithi070 ceph-mon[33250]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:41 smithi070 ceph-mon[33250]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:18:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:41 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:18:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:41 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:18:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:41 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:41.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:41 smithi067 ceph-mon[29731]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:41.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:41 smithi067 ceph-mon[29731]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:18:41.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:41 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:18:41.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:41 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:18:41.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:41 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:41.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:41 smithi082 ceph-mon[33535]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:41.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:41 smithi082 ceph-mon[33535]: from='client.24320 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi070:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:18:41.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:41 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:18:41.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:41 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:18:41.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:41 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:43.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:43 smithi067 ceph-mon[29731]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:43.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:43 smithi067 ceph-mon[29731]: from='client.? 172.21.15.70:0/3679572228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec4cffbc-0127-47e5-8176-0972f74a5514"}]: dispatch 2024-05-03T17:18:43.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:43 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec4cffbc-0127-47e5-8176-0972f74a5514"}]: dispatch 2024-05-03T17:18:43.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:43 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec4cffbc-0127-47e5-8176-0972f74a5514"}]': finished 2024-05-03T17:18:43.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:43 smithi067 ceph-mon[29731]: osdmap e62: 11 total, 10 up, 11 in 2024-05-03T17:18:43.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:18:43.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:43 smithi082 ceph-mon[33535]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:43 smithi082 ceph-mon[33535]: from='client.? 172.21.15.70:0/3679572228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec4cffbc-0127-47e5-8176-0972f74a5514"}]: dispatch 2024-05-03T17:18:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:43 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec4cffbc-0127-47e5-8176-0972f74a5514"}]: dispatch 2024-05-03T17:18:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:43 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec4cffbc-0127-47e5-8176-0972f74a5514"}]': finished 2024-05-03T17:18:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:43 smithi082 ceph-mon[33535]: osdmap e62: 11 total, 10 up, 11 in 2024-05-03T17:18:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:18:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:43 smithi070 ceph-mon[33250]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:43 smithi070 ceph-mon[33250]: from='client.? 172.21.15.70:0/3679572228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec4cffbc-0127-47e5-8176-0972f74a5514"}]: dispatch 2024-05-03T17:18:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:43 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ec4cffbc-0127-47e5-8176-0972f74a5514"}]: dispatch 2024-05-03T17:18:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:43 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ec4cffbc-0127-47e5-8176-0972f74a5514"}]': finished 2024-05-03T17:18:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:43 smithi070 ceph-mon[33250]: osdmap e62: 11 total, 10 up, 11 in 2024-05-03T17:18:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:18:44.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:44 smithi067 ceph-mon[29731]: from='client.? 172.21.15.70:0/2499462647' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:18:44.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:44 smithi082 ceph-mon[33535]: from='client.? 172.21.15.70:0/2499462647' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:18:44.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:44 smithi070 ceph-mon[33250]: from='client.? 172.21.15.70:0/2499462647' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:18:45.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:45 smithi082 ceph-mon[33535]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:45.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:45 smithi070 ceph-mon[33250]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:45.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:45 smithi067 ceph-mon[29731]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:47.662 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:47 smithi070 ceph-mon[33250]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:47.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:47 smithi082 ceph-mon[33535]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:47 smithi067 ceph-mon[29731]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:49.543 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:49 smithi070 ceph-mon[33250]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:49.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:49 smithi082 ceph-mon[33535]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:49.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:49 smithi067 ceph-mon[29731]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:51.589 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:51 smithi070 ceph-mon[33250]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:51.589 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:51 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-03T17:18:51.589 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:51 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:51.589 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:51 smithi070 ceph-mon[33250]: Deploying daemon osd.10 on smithi070 2024-05-03T17:18:51.756 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:51 smithi067 ceph-mon[29731]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:51.756 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:51 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-03T17:18:51.756 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:51 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:51.756 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:51 smithi067 ceph-mon[29731]: Deploying daemon osd.10 on smithi070 2024-05-03T17:18:51.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:51 smithi082 ceph-mon[33535]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:51.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:51 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-03T17:18:51.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:51 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:18:51.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:51 smithi082 ceph-mon[33535]: Deploying daemon osd.10 on smithi070 2024-05-03T17:18:53.633 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:53 smithi070 ceph-mon[33250]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:53.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:53 smithi082 ceph-mon[33535]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:53.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:53 smithi067 ceph-mon[29731]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:55.693 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:55 smithi070 ceph-mon[33250]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:55.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:55 smithi082 ceph-mon[33535]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:55.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:55 smithi067 ceph-mon[29731]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:57.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:57 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:57.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:57 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:57.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:57 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:57.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:57 smithi082 ceph-mon[33535]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:57.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:57 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:57.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:57 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:57.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:57 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:57.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:57 smithi070 ceph-mon[33250]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:57.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:57 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:57.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:57 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:18:57.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:57 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:18:57.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:57 smithi067 ceph-mon[29731]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:59.499 INFO:teuthology.orchestra.run.smithi070.stdout:Created osd(s) 10 on host 'smithi070' 2024-05-03T17:18:59.499 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.498+0000 7fa4977fe700 1 -- 172.21.15.70:0/1289575419 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7fa4a00639a0 con 0x7fa48c0b2900 2024-05-03T17:18:59.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.500+0000 7fa4a82b3700 1 -- 172.21.15.70:0/1289575419 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa48c0b2900 msgr2=0x7fa48c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:18:59.501 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.500+0000 7fa4a82b3700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa48c0b2900 0x7fa48c0b4dc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0104560 tx=0x7fa49c008040 comp rx=0 tx=0).stop 2024-05-03T17:18:59.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.501+0000 7fa4a82b3700 1 -- 172.21.15.70:0/1289575419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4a01050a0 msgr2=0x7fa4a01a0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:18:59.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.501+0000 7fa4a82b3700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4a01050a0 0x7fa4a01a0d20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa498009fd0 tx=0x7fa498004e40 comp rx=0 tx=0).stop 2024-05-03T17:18:59.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.501+0000 7fa4a82b3700 1 -- 172.21.15.70:0/1289575419 shutdown_connections 2024-05-03T17:18:59.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.501+0000 7fa4a82b3700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa48c0b2900 0x7fa48c0b4dc0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:59.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.501+0000 7fa4a82b3700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa4a01050a0 0x7fa4a01a0d20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:59.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.501+0000 7fa4a82b3700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa4a0103500 0x7fa4a0199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:59.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.501+0000 7fa4a82b3700 1 --2- 172.21.15.70:0/1289575419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa4a0104700 0x7fa4a0199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:18:59.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.501+0000 7fa4a82b3700 1 -- 172.21.15.70:0/1289575419 >> 172.21.15.70:0/1289575419 conn(0x7fa4a00feaa0 msgr2=0x7fa4a0107b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:18:59.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.501+0000 7fa4a82b3700 1 -- 172.21.15.70:0/1289575419 shutdown_connections 2024-05-03T17:18:59.502 INFO:teuthology.orchestra.run.smithi070.stderr:2024-05-03T17:18:59.501+0000 7fa4a82b3700 1 -- 172.21.15.70:0/1289575419 wait complete. 2024-05-03T17:18:59.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:18:59 smithi067 ceph-mon[29731]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:59.741 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:18:59 smithi070 ceph-mon[33250]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:18:59.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:18:59 smithi082 ceph-mon[33535]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:19:00.374 DEBUG:teuthology.orchestra.run.smithi070:osd.10> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.10.service 2024-05-03T17:19:00.376 INFO:tasks.cephadm:Deploying osd.11 on smithi082 with /dev/vg_nvme/lv_1... 2024-05-03T17:19:00.376 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-03T17:19:00.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:00 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:00.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:00 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:00.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:00 smithi082 ceph-mon[33535]: from='osd.10 [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-03T17:19:00.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:00 smithi082 ceph-mon[33535]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-03T17:19:00.618 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:19:00.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:00 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:00.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:00 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:00.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:00 smithi070 ceph-mon[33250]: from='osd.10 [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-03T17:19:00.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:00 smithi070 ceph-mon[33250]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-03T17:19:00.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:00 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:00.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:00 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:00.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:00 smithi067 ceph-mon[29731]: from='osd.10 [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-03T17:19:00.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:00 smithi067 ceph-mon[29731]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-03T17:19:01.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:01 smithi070 ceph-mon[33250]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:19:01.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:01 smithi070 ceph-mon[33250]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-03T17:19:01.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:01 smithi070 ceph-mon[33250]: osdmap e63: 11 total, 10 up, 11 in 2024-05-03T17:19:01.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:01 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:19:01.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:01 smithi070 ceph-mon[33250]: from='osd.10 [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:19:01.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:01 smithi070 ceph-mon[33250]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:19:01.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:01 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:01.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:01 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:01.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:01 smithi082 ceph-mon[33535]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:19:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:01 smithi082 ceph-mon[33535]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-03T17:19:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:01 smithi082 ceph-mon[33535]: osdmap e63: 11 total, 10 up, 11 in 2024-05-03T17:19:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:01 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:19:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:01 smithi082 ceph-mon[33535]: from='osd.10 [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:19:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:01 smithi082 ceph-mon[33535]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:19:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:01 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:01 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:01.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:01 smithi067 ceph-mon[29731]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:19:01.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:01 smithi067 ceph-mon[29731]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-03T17:19:01.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:01 smithi067 ceph-mon[29731]: osdmap e63: 11 total, 10 up, 11 in 2024-05-03T17:19:01.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:01 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:19:01.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:01 smithi067 ceph-mon[29731]: from='osd.10 [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:19:01.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:01 smithi067 ceph-mon[29731]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]: dispatch 2024-05-03T17:19:01.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:01 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:01.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:01 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:02.163 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:19:01 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:19:01.909+0000 7f98a4fed700 -1 osd.10 0 waiting for initial osdmap 2024-05-03T17:19:02.163 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:19:01 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:19:01.920+0000 7f989cdd8700 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T17:19:02.750 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:02 smithi070 ceph-mon[33250]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-05-03T17:19:02.750 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:02 smithi070 ceph-mon[33250]: osdmap e64: 11 total, 10 up, 11 in 2024-05-03T17:19:02.750 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:02 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:19:02.750 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:02 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:19:02.750 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:02 smithi070 ceph-mon[33250]: from='osd.10 ' entity='osd.10' 2024-05-03T17:19:02.750 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:02 smithi070 ceph-mon[33250]: osd.10 [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629] boot 2024-05-03T17:19:02.750 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:02 smithi070 ceph-mon[33250]: osdmap e65: 11 total, 11 up, 11 in 2024-05-03T17:19:02.750 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:02 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:19:02.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:02 smithi082 ceph-mon[33535]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-05-03T17:19:02.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:02 smithi082 ceph-mon[33535]: osdmap e64: 11 total, 10 up, 11 in 2024-05-03T17:19:02.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:02 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:19:02.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:02 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:19:02.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:02 smithi082 ceph-mon[33535]: from='osd.10 ' entity='osd.10' 2024-05-03T17:19:02.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:02 smithi082 ceph-mon[33535]: osd.10 [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629] boot 2024-05-03T17:19:02.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:02 smithi082 ceph-mon[33535]: osdmap e65: 11 total, 11 up, 11 in 2024-05-03T17:19:02.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:02 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:19:02.829 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-03T17:19:02.829 INFO:teuthology.orchestra.run.smithi082.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-03T17:19:02.829 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10+0 records in 2024-05-03T17:19:02.829 INFO:teuthology.orchestra.run.smithi082.stderr:10+0 records out 2024-05-03T17:19:02.829 INFO:teuthology.orchestra.run.smithi082.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0118985 s, 881 MB/s 2024-05-03T17:19:02.829 INFO:teuthology.orchestra.run.smithi082.stderr:--> Zapping successful for: 2024-05-03T17:19:02.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:02 smithi067 ceph-mon[29731]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi070", "root=default"]}]': finished 2024-05-03T17:19:02.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:02 smithi067 ceph-mon[29731]: osdmap e64: 11 total, 10 up, 11 in 2024-05-03T17:19:02.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:02 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:19:02.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:02 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:19:02.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:02 smithi067 ceph-mon[29731]: from='osd.10 ' entity='osd.10' 2024-05-03T17:19:02.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:02 smithi067 ceph-mon[29731]: osd.10 [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629] boot 2024-05-03T17:19:02.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:02 smithi067 ceph-mon[29731]: osdmap e65: 11 total, 11 up, 11 in 2024-05-03T17:19:02.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:02 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-03T17:19:03.659 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch daemon add osd smithi082:vg_nvme/lv_1 2024-05-03T17:19:03.906 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:19:04.207 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:03 smithi082 ceph-mon[33535]: purged_snaps scrub starts 2024-05-03T17:19:04.207 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:03 smithi082 ceph-mon[33535]: purged_snaps scrub ok 2024-05-03T17:19:04.207 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:03 smithi082 ceph-mon[33535]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:19:04.207 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:03 smithi082 ceph-mon[33535]: osdmap e66: 11 total, 11 up, 11 in 2024-05-03T17:19:04.296 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:03 smithi070 ceph-mon[33250]: purged_snaps scrub starts 2024-05-03T17:19:04.296 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:03 smithi070 ceph-mon[33250]: purged_snaps scrub ok 2024-05-03T17:19:04.296 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:03 smithi070 ceph-mon[33250]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:19:04.297 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:03 smithi070 ceph-mon[33250]: osdmap e66: 11 total, 11 up, 11 in 2024-05-03T17:19:04.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:03 smithi067 ceph-mon[29731]: purged_snaps scrub starts 2024-05-03T17:19:04.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:03 smithi067 ceph-mon[29731]: purged_snaps scrub ok 2024-05-03T17:19:04.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:03 smithi067 ceph-mon[29731]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 286 MiB used, 894 GiB / 894 GiB avail 2024-05-03T17:19:04.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:03 smithi067 ceph-mon[29731]: osdmap e66: 11 total, 11 up, 11 in 2024-05-03T17:19:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:04 smithi082 ceph-mon[33535]: osdmap e67: 11 total, 11 up, 11 in 2024-05-03T17:19:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:05.266 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:04 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:19:05.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:04 smithi070 ceph-mon[33250]: osdmap e67: 11 total, 11 up, 11 in 2024-05-03T17:19:05.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:05.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:05.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:05.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:05.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:05.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:05.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:05.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:05.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:05.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:05.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:04 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:19:05.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:04 smithi067 ceph-mon[29731]: osdmap e67: 11 total, 11 up, 11 in 2024-05-03T17:19:05.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:04 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:19:05.823 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.821+0000 7f91b1de7700 1 -- 172.21.15.82:0/1638237805 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91ac103500 msgr2=0x7f91ac103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:05.823 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.821+0000 7f91b1de7700 1 --2- 172.21.15.82:0/1638237805 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91ac103500 0x7f91ac103920 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f919c009990 tx=0x7f919c009ca0 comp rx=0 tx=0).stop 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.821+0000 7f91b1de7700 1 -- 172.21.15.82:0/1638237805 shutdown_connections 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.821+0000 7f91b1de7700 1 --2- 172.21.15.82:0/1638237805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac1050a0 0x7f91ac10b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.821+0000 7f91b1de7700 1 --2- 172.21.15.82:0/1638237805 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91ac103500 0x7f91ac103920 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.821+0000 7f91b1de7700 1 --2- 172.21.15.82:0/1638237805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91ac104700 0x7f91ac104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.821+0000 7f91b1de7700 1 -- 172.21.15.82:0/1638237805 >> 172.21.15.82:0/1638237805 conn(0x7f91ac0feaa0 msgr2=0x7f91ac100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.821+0000 7f91b1de7700 1 -- 172.21.15.82:0/1638237805 shutdown_connections 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.821+0000 7f91b1de7700 1 -- 172.21.15.82:0/1638237805 wait complete. 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.821+0000 7f91b1de7700 1 Processor -- start 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.821+0000 7f91b1de7700 1 -- start start 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91b1de7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91ac103500 0x7f91ac199740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91b1de7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91ac104700 0x7f91ac199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91b1de7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac1050a0 0x7f91ac1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91b1de7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f91ac10eff0 con 0x7f91ac1050a0 2024-05-03T17:19:05.824 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91b1de7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f91ac10ee70 con 0x7f91ac103500 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91b1de7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f91ac10f170 con 0x7f91ac104700 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91a2ffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91ac104700 0x7f91ac199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91a2ffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91ac104700 0x7f91ac199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:38732/0 (socket says 172.21.15.82:38732) 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91a2ffd700 1 -- 172.21.15.82:0/1457413651 learned_addr learned my addr 172.21.15.82:0/1457413651 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91ab7fe700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91ac103500 0x7f91ac199740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91abfff700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac1050a0 0x7f91ac1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91a2ffd700 1 -- 172.21.15.82:0/1457413651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91ac103500 msgr2=0x7f91ac199740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91a2ffd700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91ac103500 0x7f91ac199740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91a2ffd700 1 -- 172.21.15.82:0/1457413651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac1050a0 msgr2=0x7f91ac1a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91a2ffd700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac1050a0 0x7f91ac1a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91a2ffd700 1 -- 172.21.15.82:0/1457413651 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91ac1a13a0 con 0x7f91ac104700 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91abfff700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac1050a0 0x7f91ac1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91a2ffd700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91ac104700 0x7f91ac199c80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f919400c9a0 tx=0x7f919400ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91a97fa700 1 -- 172.21.15.82:0/1457413651 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194007a60 con 0x7f91ac104700 2024-05-03T17:19:05.825 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91b1de7700 1 -- 172.21.15.82:0/1457413651 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91ac1a1690 con 0x7f91ac104700 2024-05-03T17:19:05.826 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91a97fa700 1 -- 172.21.15.82:0/1457413651 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9194004ce0 con 0x7f91ac104700 2024-05-03T17:19:05.826 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91a97fa700 1 -- 172.21.15.82:0/1457413651 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9194005560 con 0x7f91ac104700 2024-05-03T17:19:05.826 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.822+0000 7f91b1de7700 1 -- 172.21.15.82:0/1457413651 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91ac1a1b80 con 0x7f91ac104700 2024-05-03T17:19:05.827 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.824+0000 7f91b1de7700 1 -- 172.21.15.82:0/1457413651 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91ac06a190 con 0x7f91ac104700 2024-05-03T17:19:05.827 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.825+0000 7f91a97fa700 1 -- 172.21.15.82:0/1457413651 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f9194007c00 con 0x7f91ac104700 2024-05-03T17:19:05.827 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.825+0000 7f91a97fa700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f91980b2880 0x7f91980b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:05.827 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.825+0000 7f91a97fa700 1 -- 172.21.15.82:0/1457413651 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6694+0+0 (secure 0 0 0) 0x7f91940e1910 con 0x7f91ac104700 2024-05-03T17:19:05.827 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.825+0000 7f91ab7fe700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f91980b2880 0x7f91980b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:05.827 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.826+0000 7f91ab7fe700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f91980b2880 0x7f91980b4d40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f919c0038a0 tx=0x7f919c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:05.830 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:05.830+0000 7f91a97fa700 1 -- 172.21.15.82:0/1457413651 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f91940a9d10 con 0x7f91ac104700 2024-05-03T17:19:06.042 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:06.042+0000 7f91b1de7700 1 -- 172.21.15.82:0/1457413651 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f91ac0638e0 con 0x7f91980b2880 2024-05-03T17:19:06.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:05 smithi082 ceph-mon[33535]: Detected new or changed devices on smithi070 2024-05-03T17:19:06.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:05 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi070 to 4272M 2024-05-03T17:19:06.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:05 smithi082 ceph-mon[33535]: pgmap v236: 1 pgs: 1 activating+degraded; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T17:19:06.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:06.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:06.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:06.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:05 smithi082 ceph-mon[33535]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-05-03T17:19:06.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:05 smithi070 ceph-mon[33250]: Detected new or changed devices on smithi070 2024-05-03T17:19:06.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:05 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi070 to 4272M 2024-05-03T17:19:06.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:05 smithi070 ceph-mon[33250]: pgmap v236: 1 pgs: 1 activating+degraded; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T17:19:06.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:06.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:06.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:06.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:05 smithi070 ceph-mon[33250]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-05-03T17:19:06.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:05 smithi067 ceph-mon[29731]: Detected new or changed devices on smithi070 2024-05-03T17:19:06.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:05 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi070 to 4272M 2024-05-03T17:19:06.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:05 smithi067 ceph-mon[29731]: pgmap v236: 1 pgs: 1 activating+degraded; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T17:19:06.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:06.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:06.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:06.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:05 smithi067 ceph-mon[29731]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-05-03T17:19:07.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:06 smithi082 ceph-mon[33535]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:19:07.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:19:07.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:19:07.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:07.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:06 smithi070 ceph-mon[33250]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:19:07.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:19:07.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:19:07.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:07.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:06 smithi067 ceph-mon[29731]: from='client.24340 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi082:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:19:07.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-03T17:19:07.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-03T17:19:07.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:08.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:07 smithi082 ceph-mon[33535]: pgmap v237: 1 pgs: 1 activating+degraded; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T17:19:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:07 smithi070 ceph-mon[33250]: pgmap v237: 1 pgs: 1 activating+degraded; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T17:19:08.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:07 smithi067 ceph-mon[29731]: pgmap v237: 1 pgs: 1 activating+degraded; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T17:19:09.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:08 smithi082 ceph-mon[33535]: from='client.? 172.21.15.82:0/420370417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8acfb0b-98e8-49e1-a114-f33aa8160b62"}]: dispatch 2024-05-03T17:19:09.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:08 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8acfb0b-98e8-49e1-a114-f33aa8160b62"}]: dispatch 2024-05-03T17:19:09.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:08 smithi082 ceph-mon[33535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a8acfb0b-98e8-49e1-a114-f33aa8160b62"}]': finished 2024-05-03T17:19:09.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:08 smithi082 ceph-mon[33535]: osdmap e68: 12 total, 11 up, 12 in 2024-05-03T17:19:09.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:08 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:08 smithi070 ceph-mon[33250]: from='client.? 172.21.15.82:0/420370417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8acfb0b-98e8-49e1-a114-f33aa8160b62"}]: dispatch 2024-05-03T17:19:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:08 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8acfb0b-98e8-49e1-a114-f33aa8160b62"}]: dispatch 2024-05-03T17:19:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:08 smithi070 ceph-mon[33250]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a8acfb0b-98e8-49e1-a114-f33aa8160b62"}]': finished 2024-05-03T17:19:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:08 smithi070 ceph-mon[33250]: osdmap e68: 12 total, 11 up, 12 in 2024-05-03T17:19:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:08 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:09.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:08 smithi067 ceph-mon[29731]: from='client.? 172.21.15.82:0/420370417' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8acfb0b-98e8-49e1-a114-f33aa8160b62"}]: dispatch 2024-05-03T17:19:09.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:08 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a8acfb0b-98e8-49e1-a114-f33aa8160b62"}]: dispatch 2024-05-03T17:19:09.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:08 smithi067 ceph-mon[29731]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a8acfb0b-98e8-49e1-a114-f33aa8160b62"}]': finished 2024-05-03T17:19:09.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:08 smithi067 ceph-mon[29731]: osdmap e68: 12 total, 11 up, 12 in 2024-05-03T17:19:09.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:08 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:10.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:09 smithi082 ceph-mon[33535]: pgmap v238: 1 pgs: 1 activating+degraded; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T17:19:10.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:09 smithi082 ceph-mon[33535]: from='client.? 172.21.15.82:0/2892566444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:19:10.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:09 smithi070 ceph-mon[33250]: pgmap v238: 1 pgs: 1 activating+degraded; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T17:19:10.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:09 smithi070 ceph-mon[33250]: from='client.? 172.21.15.82:0/2892566444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:19:10.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:09 smithi067 ceph-mon[29731]: pgmap v238: 1 pgs: 1 activating+degraded; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2024-05-03T17:19:10.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:09 smithi067 ceph-mon[29731]: from='client.? 172.21.15.82:0/2892566444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-03T17:19:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:11 smithi070 ceph-mon[33250]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-05-03T17:19:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:11 smithi070 ceph-mon[33250]: Cluster is now healthy 2024-05-03T17:19:11.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:11 smithi067 ceph-mon[29731]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-05-03T17:19:11.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:11 smithi067 ceph-mon[29731]: Cluster is now healthy 2024-05-03T17:19:11.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:11 smithi082 ceph-mon[33535]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-05-03T17:19:11.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:11 smithi082 ceph-mon[33535]: Cluster is now healthy 2024-05-03T17:19:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:12 smithi070 ceph-mon[33250]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-05-03T17:19:12.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:12 smithi067 ceph-mon[29731]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-05-03T17:19:12.494 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:12 smithi082 ceph-mon[33535]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-05-03T17:19:14.216 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:14 smithi082 ceph-mon[33535]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-03T17:19:14.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:14 smithi070 ceph-mon[33250]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-03T17:19:14.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:14 smithi067 ceph-mon[29731]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-05-03T17:19:16.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:16 smithi070 ceph-mon[33250]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:19:16.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:16 smithi067 ceph-mon[29731]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:19:16.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:16 smithi082 ceph-mon[33535]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:19:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-03T17:19:17.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:17.384 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-03T17:19:17.384 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:17.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-03T17:19:17.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:18.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:18 smithi070 ceph-mon[33250]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:19:18.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:18 smithi070 ceph-mon[33250]: Deploying daemon osd.11 on smithi082 2024-05-03T17:19:18.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:18 smithi067 ceph-mon[29731]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:19:18.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:18 smithi067 ceph-mon[29731]: Deploying daemon osd.11 on smithi082 2024-05-03T17:19:18.476 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:18 smithi082 ceph-mon[33535]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:19:18.476 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:18 smithi082 ceph-mon[33535]: Deploying daemon osd.11 on smithi082 2024-05-03T17:19:20.198 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:20 smithi082 ceph-mon[33535]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:19:20.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:20 smithi070 ceph-mon[33250]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:19:20.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:20 smithi067 ceph-mon[29731]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-03T17:19:22.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:22 smithi070 ceph-mon[33250]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-05-03T17:19:22.410 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:22 smithi082 ceph-mon[33535]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-05-03T17:19:22.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:22 smithi067 ceph-mon[29731]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-05-03T17:19:23.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:23 smithi067 ceph-mon[29731]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-03T17:19:23.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:23.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:23.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:19:24.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:23 smithi082 ceph-mon[33535]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-03T17:19:24.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:24.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:24.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:19:24.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:23 smithi070 ceph-mon[33250]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-03T17:19:24.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:24.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:24.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:19:25.940 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:25 smithi082 ceph-mon[33535]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-03T17:19:25.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:25 smithi067 ceph-mon[29731]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-03T17:19:26.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:25 smithi070 ceph-mon[33250]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-03T17:19:26.312 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.310+0000 7f91a97fa700 1 -- 172.21.15.82:0/1457413651 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f91ac0638e0 con 0x7f91980b2880 2024-05-03T17:19:26.312 INFO:teuthology.orchestra.run.smithi082.stdout:Created osd(s) 11 on host 'smithi082' 2024-05-03T17:19:26.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.313+0000 7f91b1de7700 1 -- 172.21.15.82:0/1457413651 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f91980b2880 msgr2=0x7f91980b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:26.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.313+0000 7f91b1de7700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f91980b2880 0x7f91980b4d40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f919c0038a0 tx=0x7f919c005fb0 comp rx=0 tx=0).stop 2024-05-03T17:19:26.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.313+0000 7f91b1de7700 1 -- 172.21.15.82:0/1457413651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91ac104700 msgr2=0x7f91ac199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:26.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.313+0000 7f91b1de7700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91ac104700 0x7f91ac199c80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f919400c9a0 tx=0x7f919400ccb0 comp rx=0 tx=0).stop 2024-05-03T17:19:26.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.313+0000 7f91b1de7700 1 -- 172.21.15.82:0/1457413651 shutdown_connections 2024-05-03T17:19:26.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.313+0000 7f91b1de7700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f91980b2880 0x7f91980b4d40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:26.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.313+0000 7f91b1de7700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f91ac1050a0 0x7f91ac1a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:26.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.313+0000 7f91b1de7700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f91ac103500 0x7f91ac199740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:26.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.313+0000 7f91b1de7700 1 --2- 172.21.15.82:0/1457413651 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f91ac104700 0x7f91ac199c80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:26.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.313+0000 7f91b1de7700 1 -- 172.21.15.82:0/1457413651 >> 172.21.15.82:0/1457413651 conn(0x7f91ac0feaa0 msgr2=0x7f91ac107b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:26.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.313+0000 7f91b1de7700 1 -- 172.21.15.82:0/1457413651 shutdown_connections 2024-05-03T17:19:26.315 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:26.313+0000 7f91b1de7700 1 -- 172.21.15.82:0/1457413651 wait complete. 2024-05-03T17:19:27.004 DEBUG:teuthology.orchestra.run.smithi082:osd.11> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.11.service 2024-05-03T17:19:27.006 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-03T17:19:27.007 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd stat -f json 2024-05-03T17:19:27.267 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:19:27.543 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:27 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:27.543 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:27 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:27.543 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:27 smithi067 ceph-mon[29731]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-03T17:19:27.543 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:27 smithi067 ceph-mon[29731]: from='osd.11 [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-03T17:19:27.543 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:27 smithi067 ceph-mon[29731]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-03T17:19:27.543 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:27 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:27.543 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:27 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:27.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:27 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:27.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:27 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:27.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:27 smithi070 ceph-mon[33250]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-03T17:19:27.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:27 smithi070 ceph-mon[33250]: from='osd.11 [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-03T17:19:27.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:27 smithi070 ceph-mon[33250]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-03T17:19:27.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:27 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:27.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:27 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:27.730 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:27 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:27.731 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:27 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:27.731 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:27 smithi082 ceph-mon[33535]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-03T17:19:27.731 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:27 smithi082 ceph-mon[33535]: from='osd.11 [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-03T17:19:27.731 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:27 smithi082 ceph-mon[33535]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-03T17:19:27.731 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:27 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:27.731 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:27 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:28.651 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:28 smithi067 ceph-mon[29731]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-03T17:19:28.651 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:28 smithi067 ceph-mon[29731]: osdmap e69: 12 total, 11 up, 12 in 2024-05-03T17:19:28.651 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:28 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:28.651 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:28 smithi067 ceph-mon[29731]: from='osd.11 [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:19:28.652 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:28 smithi067 ceph-mon[29731]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:19:28.750 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:28 smithi082 ceph-mon[33535]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-03T17:19:28.751 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:28 smithi082 ceph-mon[33535]: osdmap e69: 12 total, 11 up, 12 in 2024-05-03T17:19:28.751 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:28 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:28.751 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:28 smithi082 ceph-mon[33535]: from='osd.11 [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:19:28.751 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:28 smithi082 ceph-mon[33535]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:19:28.751 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:19:28 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:19:28.307+0000 7f594d757700 -1 osd.11 0 waiting for initial osdmap 2024-05-03T17:19:28.752 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:19:28 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:19:28.320+0000 7f594954a700 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-03T17:19:28.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:28 smithi070 ceph-mon[33250]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-03T17:19:28.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:28 smithi070 ceph-mon[33250]: osdmap e69: 12 total, 11 up, 12 in 2024-05-03T17:19:28.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:28 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:28.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:28 smithi070 ceph-mon[33250]: from='osd.11 [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:19:28.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:28 smithi070 ceph-mon[33250]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]: dispatch 2024-05-03T17:19:29.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.008+0000 7ff0775c8700 1 -- 172.21.15.67:0/3434898723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff070103500 msgr2=0x7ff070103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:29.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.008+0000 7ff0775c8700 1 --2- 172.21.15.67:0/3434898723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff070103500 0x7ff070103920 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7ff0600099c0 tx=0x7ff060009cd0 comp rx=0 tx=0).stop 2024-05-03T17:19:29.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.008+0000 7ff0775c8700 1 -- 172.21.15.67:0/3434898723 shutdown_connections 2024-05-03T17:19:29.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.008+0000 7ff0775c8700 1 --2- 172.21.15.67:0/3434898723 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff0701050a0 0x7ff07010b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:29.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.008+0000 7ff0775c8700 1 --2- 172.21.15.67:0/3434898723 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff070103500 0x7ff070103920 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:29.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.008+0000 7ff0775c8700 1 --2- 172.21.15.67:0/3434898723 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff070104700 0x7ff070104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.008+0000 7ff0775c8700 1 -- 172.21.15.67:0/3434898723 >> 172.21.15.67:0/3434898723 conn(0x7ff0700feaa0 msgr2=0x7ff070100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.008+0000 7ff0775c8700 1 -- 172.21.15.67:0/3434898723 shutdown_connections 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.008+0000 7ff0775c8700 1 -- 172.21.15.67:0/3434898723 wait complete. 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff0775c8700 1 Processor -- start 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff0775c8700 1 -- start start 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff0775c8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff070103500 0x7ff070199600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff0775c8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff070104700 0x7ff070199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff0775c8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0701050a0 0x7ff0701a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff0775c8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff07010ef90 con 0x7ff070103500 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff0775c8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff07010ee10 con 0x7ff0701050a0 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff0775c8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff07010f110 con 0x7ff070104700 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff075364700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff070103500 0x7ff070199600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:29.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff075364700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff070103500 0x7ff070199600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39796/0 (socket says 172.21.15.67:39796) 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff075364700 1 -- 172.21.15.67:0/2742984957 learned_addr learned my addr 172.21.15.67:0/2742984957 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff075b65700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0701050a0 0x7ff0701a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.009+0000 7ff074b63700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff070104700 0x7ff070199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.010+0000 7ff075364700 1 -- 172.21.15.67:0/2742984957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff070104700 msgr2=0x7ff070199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.010+0000 7ff075364700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff070104700 0x7ff070199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.010+0000 7ff075364700 1 -- 172.21.15.67:0/2742984957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0701050a0 msgr2=0x7ff0701a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.010+0000 7ff075364700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0701050a0 0x7ff0701a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.010+0000 7ff075364700 1 -- 172.21.15.67:0/2742984957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0701a1260 con 0x7ff070103500 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.010+0000 7ff075364700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff070103500 0x7ff070199600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff06000b580 tx=0x7ff0600057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.010+0000 7ff0667fc700 1 -- 172.21.15.67:0/2742984957 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff060027070 con 0x7ff070103500 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.010+0000 7ff0667fc700 1 -- 172.21.15.67:0/2742984957 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff0600040e0 con 0x7ff070103500 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.010+0000 7ff0667fc700 1 -- 172.21.15.67:0/2742984957 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff060022d40 con 0x7ff070103500 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.010+0000 7ff0775c8700 1 -- 172.21.15.67:0/2742984957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0701a14f0 con 0x7ff070103500 2024-05-03T17:19:29.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.010+0000 7ff0775c8700 1 -- 172.21.15.67:0/2742984957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0701a1a20 con 0x7ff070103500 2024-05-03T17:19:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.011+0000 7ff0775c8700 1 -- 172.21.15.67:0/2742984957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff07006a190 con 0x7ff070103500 2024-05-03T17:19:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.012+0000 7ff0667fc700 1 -- 172.21.15.67:0/2742984957 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ff06002a070 con 0x7ff070103500 2024-05-03T17:19:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.013+0000 7ff0667fc700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff05c0b2900 0x7ff05c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.013+0000 7ff074b63700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff05c0b2900 0x7ff05c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.013+0000 7ff0667fc700 1 -- 172.21.15.67:0/2742984957 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6847+0+0 (secure 0 0 0) 0x7ff0600e2f90 con 0x7ff070103500 2024-05-03T17:19:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.013+0000 7ff074b63700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff05c0b2900 0x7ff05c0b4dc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff06c005950 tx=0x7ff06c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:29.017 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.016+0000 7ff0667fc700 1 -- 172.21.15.67:0/2742984957 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ff060031020 con 0x7ff070103500 2024-05-03T17:19:29.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.231+0000 7ff0775c8700 1 -- 172.21.15.67:0/2742984957 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7ff0700640b0 con 0x7ff070103500 2024-05-03T17:19:29.232 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.231+0000 7ff0667fc700 1 -- 172.21.15.67:0/2742984957 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) ==== 74+0+133 (secure 0 0 0) 0x7ff0600cbe20 con 0x7ff070103500 2024-05-03T17:19:29.232 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:19:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.233+0000 7ff0775c8700 1 -- 172.21.15.67:0/2742984957 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff05c0b2900 msgr2=0x7ff05c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.233+0000 7ff0775c8700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff05c0b2900 0x7ff05c0b4dc0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff06c005950 tx=0x7ff06c00b410 comp rx=0 tx=0).stop 2024-05-03T17:19:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.233+0000 7ff0775c8700 1 -- 172.21.15.67:0/2742984957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff070103500 msgr2=0x7ff070199600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.233+0000 7ff0775c8700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff070103500 0x7ff070199600 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff06000b580 tx=0x7ff0600057a0 comp rx=0 tx=0).stop 2024-05-03T17:19:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.233+0000 7ff0775c8700 1 -- 172.21.15.67:0/2742984957 shutdown_connections 2024-05-03T17:19:29.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.233+0000 7ff0775c8700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff05c0b2900 0x7ff05c0b4dc0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.233+0000 7ff0775c8700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff0701050a0 0x7ff0701a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.233+0000 7ff0775c8700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff070103500 0x7ff070199600 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.233+0000 7ff0775c8700 1 --2- 172.21.15.67:0/2742984957 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff070104700 0x7ff070199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.233+0000 7ff0775c8700 1 -- 172.21.15.67:0/2742984957 >> 172.21.15.67:0/2742984957 conn(0x7ff0700feaa0 msgr2=0x7ff070107a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.233+0000 7ff0775c8700 1 -- 172.21.15.67:0/2742984957 shutdown_connections 2024-05-03T17:19:29.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:29.233+0000 7ff0775c8700 1 -- 172.21.15.67:0/2742984957 wait complete. 2024-05-03T17:19:29.337 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:29 smithi082 ceph-mon[33535]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T17:19:29.337 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:29 smithi082 ceph-mon[33535]: osdmap e70: 12 total, 11 up, 12 in 2024-05-03T17:19:29.337 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:29 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:29.338 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:29 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:29.338 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:29 smithi082 ceph-mon[33535]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-03T17:19:29.338 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:29 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2742984957' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T17:19:29.338 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:29 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:29.536 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:29 smithi067 ceph-mon[29731]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T17:19:29.536 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:29 smithi067 ceph-mon[29731]: osdmap e70: 12 total, 11 up, 12 in 2024-05-03T17:19:29.536 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:29 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:29.536 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:29 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:29.536 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:29 smithi067 ceph-mon[29731]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-03T17:19:29.536 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:29 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2742984957' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T17:19:29.536 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:29 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:29 smithi070 ceph-mon[33250]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi082", "root=default"]}]': finished 2024-05-03T17:19:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:29 smithi070 ceph-mon[33250]: osdmap e70: 12 total, 11 up, 12 in 2024-05-03T17:19:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:29 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:29 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:29 smithi070 ceph-mon[33250]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-05-03T17:19:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:29 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2742984957' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T17:19:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:29 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:29.928 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":70,"num_osds":12,"num_up_osds":11,"osd_up_since":1714756741,"num_in_osds":12,"osd_in_since":1714756748,"num_remapped_pgs":0} 2024-05-03T17:19:30.609 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:30 smithi082 ceph-mon[33535]: purged_snaps scrub starts 2024-05-03T17:19:30.609 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:30 smithi082 ceph-mon[33535]: purged_snaps scrub ok 2024-05-03T17:19:30.609 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:30 smithi082 ceph-mon[33535]: osd.11 [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149] boot 2024-05-03T17:19:30.609 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:30 smithi082 ceph-mon[33535]: osdmap e71: 12 total, 12 up, 12 in 2024-05-03T17:19:30.609 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:30 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:30.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:30 smithi067 ceph-mon[29731]: purged_snaps scrub starts 2024-05-03T17:19:30.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:30 smithi067 ceph-mon[29731]: purged_snaps scrub ok 2024-05-03T17:19:30.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:30 smithi067 ceph-mon[29731]: osd.11 [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149] boot 2024-05-03T17:19:30.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:30 smithi067 ceph-mon[29731]: osdmap e71: 12 total, 12 up, 12 in 2024-05-03T17:19:30.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:30 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:30.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:30 smithi070 ceph-mon[33250]: purged_snaps scrub starts 2024-05-03T17:19:30.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:30 smithi070 ceph-mon[33250]: purged_snaps scrub ok 2024-05-03T17:19:30.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:30 smithi070 ceph-mon[33250]: osd.11 [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149] boot 2024-05-03T17:19:30.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:30 smithi070 ceph-mon[33250]: osdmap e71: 12 total, 12 up, 12 in 2024-05-03T17:19:30.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:30 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-03T17:19:30.930 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd stat -f json 2024-05-03T17:19:31.175 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:19:31.629 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: osdmap e72: 12 total, 12 up, 12 in 2024-05-03T17:19:31.630 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:31.630 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: Detected new or changed devices on smithi082 2024-05-03T17:19:31.630 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.630 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.630 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:31.630 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:31.630 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:31.630 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:31.630 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi082 to 4272M 2024-05-03T17:19:31.630 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.630 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:31.630 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:31.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:19:31.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:31.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:31.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: osdmap e72: 12 total, 12 up, 12 in 2024-05-03T17:19:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: Detected new or changed devices on smithi082 2024-05-03T17:19:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:31.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi082 to 4272M 2024-05-03T17:19:31.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:31.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:31.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:19:31.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:31.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:31.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: osdmap e72: 12 total, 12 up, 12 in 2024-05-03T17:19:31.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:31.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: Detected new or changed devices on smithi082 2024-05-03T17:19:31.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:31.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:31.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:31.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:19:31.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi082 to 4272M 2024-05-03T17:19:31.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:31.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:31.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:31.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:19:31.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:31.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:31.810 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.104+0000 7fd0b09d4700 1 -- 172.21.15.67:0/465252591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0ac103620 msgr2=0x7fd0ac103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.104+0000 7fd0b09d4700 1 --2- 172.21.15.67:0/465252591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0ac103620 0x7fd0ac103a80 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0099c0 tx=0x7fd09c009cd0 comp rx=0 tx=0).stop 2024-05-03T17:19:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.105+0000 7fd0b09d4700 1 -- 172.21.15.67:0/465252591 shutdown_connections 2024-05-03T17:19:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.105+0000 7fd0b09d4700 1 --2- 172.21.15.67:0/465252591 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac103fc0 0x7fd0ac10a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:33.107 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.105+0000 7fd0b09d4700 1 --2- 172.21.15.67:0/465252591 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0ac102420 0x7fd0ac102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.105+0000 7fd0b09d4700 1 --2- 172.21.15.67:0/465252591 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0ac103620 0x7fd0ac103a80 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.105+0000 7fd0b09d4700 1 -- 172.21.15.67:0/465252591 >> 172.21.15.67:0/465252591 conn(0x7fd0ac0fda00 msgr2=0x7fd0ac0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.105+0000 7fd0b09d4700 1 -- 172.21.15.67:0/465252591 shutdown_connections 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.105+0000 7fd0b09d4700 1 -- 172.21.15.67:0/465252591 wait complete. 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.105+0000 7fd0b09d4700 1 Processor -- start 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.105+0000 7fd0b09d4700 1 -- start start 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0b09d4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0ac102420 0x7fd0ac19b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0b09d4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0ac103620 0x7fd0ac19bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0b09d4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac103fc0 0x7fd0ac1a2d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0b09d4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac10dec0 con 0x7fd0ac102420 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0b09d4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac10dd40 con 0x7fd0ac103fc0 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0b09d4700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd0ac10e040 con 0x7fd0ac103620 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0aa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0ac102420 0x7fd0ac19b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:33.108 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0aa59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0ac102420 0x7fd0ac19b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39822/0 (socket says 172.21.15.67:39822) 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0aa59c700 1 -- 172.21.15.67:0/1160477388 learned_addr learned my addr 172.21.15.67:0/1160477388 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0aad9d700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac103fc0 0x7fd0ac1a2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0a9d9b700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0ac103620 0x7fd0ac19bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0aa59c700 1 -- 172.21.15.67:0/1160477388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0ac103620 msgr2=0x7fd0ac19bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0aa59c700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0ac103620 0x7fd0ac19bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0aa59c700 1 -- 172.21.15.67:0/1160477388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac103fc0 msgr2=0x7fd0ac1a2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0aa59c700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac103fc0 0x7fd0ac1a2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0aa59c700 1 -- 172.21.15.67:0/1160477388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ac1a3430 con 0x7fd0ac102420 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0aa59c700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0ac102420 0x7fd0ac19b7d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd09400d9b0 tx=0x7fd09400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0a37fe700 1 -- 172.21.15.67:0/1160477388 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd094017070 con 0x7fd0ac102420 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0b09d4700 1 -- 172.21.15.67:0/1160477388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0ac1a3720 con 0x7fd0ac102420 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.106+0000 7fd0a37fe700 1 -- 172.21.15.67:0/1160477388 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd094009de0 con 0x7fd0ac102420 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.107+0000 7fd0a37fe700 1 -- 172.21.15.67:0/1160477388 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd09400f3c0 con 0x7fd0ac102420 2024-05-03T17:19:33.109 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.107+0000 7fd0b09d4700 1 -- 172.21.15.67:0/1160477388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0ac1a3c60 con 0x7fd0ac102420 2024-05-03T17:19:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.107+0000 7fd0b09d4700 1 -- 172.21.15.67:0/1160477388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0ac06a190 con 0x7fd0ac102420 2024-05-03T17:19:33.110 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.109+0000 7fd0a37fe700 1 -- 172.21.15.67:0/1160477388 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fd094009930 con 0x7fd0ac102420 2024-05-03T17:19:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.109+0000 7fd0a37fe700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0980b2900 0x7fd0980b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.110+0000 7fd0a9d9b700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0980b2900 0x7fd0980b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.110+0000 7fd0a37fe700 1 -- 172.21.15.67:0/1160477388 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fd0940e1770 con 0x7fd0ac102420 2024-05-03T17:19:33.111 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.110+0000 7fd0a9d9b700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0980b2900 0x7fd0980b4dc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd09c009990 tx=0x7fd09c00b870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:33.114 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.113+0000 7fd0a37fe700 1 -- 172.21.15.67:0/1160477388 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fd0940a99b0 con 0x7fd0ac102420 2024-05-03T17:19:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.325+0000 7fd0b09d4700 1 -- 172.21.15.67:0/1160477388 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7fd0ac1a40a0 con 0x7fd0ac102420 2024-05-03T17:19:33.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.325+0000 7fd0a37fe700 1 -- 172.21.15.67:0/1160477388 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7fd0940a92b0 con 0x7fd0ac102420 2024-05-03T17:19:33.329 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:19:33.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.328+0000 7fd0b09d4700 1 -- 172.21.15.67:0/1160477388 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0980b2900 msgr2=0x7fd0980b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.328+0000 7fd0b09d4700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0980b2900 0x7fd0980b4dc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd09c009990 tx=0x7fd09c00b870 comp rx=0 tx=0).stop 2024-05-03T17:19:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.328+0000 7fd0b09d4700 1 -- 172.21.15.67:0/1160477388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0ac102420 msgr2=0x7fd0ac19b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.328+0000 7fd0b09d4700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0ac102420 0x7fd0ac19b7d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd09400d9b0 tx=0x7fd09400dcc0 comp rx=0 tx=0).stop 2024-05-03T17:19:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.328+0000 7fd0b09d4700 1 -- 172.21.15.67:0/1160477388 shutdown_connections 2024-05-03T17:19:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.328+0000 7fd0b09d4700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0980b2900 0x7fd0980b4dc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.328+0000 7fd0b09d4700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0ac103fc0 0x7fd0ac1a2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.328+0000 7fd0b09d4700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd0ac102420 0x7fd0ac19b7d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.328+0000 7fd0b09d4700 1 --2- 172.21.15.67:0/1160477388 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0ac103620 0x7fd0ac19bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.328+0000 7fd0b09d4700 1 -- 172.21.15.67:0/1160477388 >> 172.21.15.67:0/1160477388 conn(0x7fd0ac0fda00 msgr2=0x7fd0ac0ff470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.328+0000 7fd0b09d4700 1 -- 172.21.15.67:0/1160477388 shutdown_connections 2024-05-03T17:19:33.331 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:33.328+0000 7fd0b09d4700 1 -- 172.21.15.67:0/1160477388 wait complete. 2024-05-03T17:19:33.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:33 smithi067 ceph-mon[29731]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:33.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:33 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1160477388' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T17:19:33.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:33 smithi082 ceph-mon[33535]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:33.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:33 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1160477388' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T17:19:33.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:33 smithi070 ceph-mon[33250]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:33.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:33 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1160477388' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-03T17:19:34.122 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1714756769,"num_in_osds":12,"osd_in_since":1714756748,"num_remapped_pgs":0} 2024-05-03T17:19:34.123 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T17:19:34.369 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:19:35.728 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:35 smithi067 ceph-mon[29731]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:35.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:35 smithi082 ceph-mon[33535]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:35.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:35 smithi070 ceph-mon[33250]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.084+0000 7f3643a8c700 1 -- 172.21.15.67:0/981731860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f363c102820 msgr2=0x7f363c110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.084+0000 7f3643a8c700 1 --2- 172.21.15.67:0/981731860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f363c102820 0x7f363c110290 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f363400b0a0 tx=0x7f363400b3b0 comp rx=0 tx=0).stop 2024-05-03T17:19:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.084+0000 7f3643a8c700 1 -- 172.21.15.67:0/981731860 shutdown_connections 2024-05-03T17:19:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.084+0000 7f3643a8c700 1 --2- 172.21.15.67:0/981731860 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f363c102820 0x7f363c110290 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.084+0000 7f3643a8c700 1 --2- 172.21.15.67:0/981731860 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f363c108580 0x7f363c108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.084+0000 7f3643a8c700 1 --2- 172.21.15.67:0/981731860 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f363c101e60 0x7f363c1022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.084+0000 7f3643a8c700 1 -- 172.21.15.67:0/981731860 >> 172.21.15.67:0/981731860 conn(0x7f363c0776c0 msgr2=0x7f363c077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.084+0000 7f3643a8c700 1 -- 172.21.15.67:0/981731860 shutdown_connections 2024-05-03T17:19:36.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.084+0000 7f3643a8c700 1 -- 172.21.15.67:0/981731860 wait complete. 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.084+0000 7f3643a8c700 1 Processor -- start 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.084+0000 7f3643a8c700 1 -- start start 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3643a8c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f363c101e60 0x7f363c103040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3643a8c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f363c102820 0x7f363c103580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3643a8c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f363c108580 0x7f363c103ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3643a8c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f363c113670 con 0x7f363c108580 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3643a8c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f363c1134f0 con 0x7f363c101e60 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3643a8c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f363c1137f0 con 0x7f363c102820 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3642029700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f363c108580 0x7f363c103ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3641828700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f363c101e60 0x7f363c103040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3641828700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f363c101e60 0x7f363c103040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.67:50090/0 (socket says 172.21.15.67:50090) 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3642029700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f363c108580 0x7f363c103ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39838/0 (socket says 172.21.15.67:39838) 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3641828700 1 -- 172.21.15.67:0/2303757565 learned_addr learned my addr 172.21.15.67:0/2303757565 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:19:36.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3641027700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f363c102820 0x7f363c103580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3642029700 1 -- 172.21.15.67:0/2303757565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f363c102820 msgr2=0x7f363c103580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3642029700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f363c102820 0x7f363c103580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3642029700 1 -- 172.21.15.67:0/2303757565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f363c101e60 msgr2=0x7f363c103040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3642029700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f363c101e60 0x7f363c103040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3642029700 1 -- 172.21.15.67:0/2303757565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f363c1a7f50 con 0x7f363c108580 2024-05-03T17:19:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3641828700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f363c101e60 0x7f363c103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:19:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.085+0000 7f3642029700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f363c108580 0x7f363c103ac0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f363400b070 tx=0x7f3634004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.086+0000 7f3632ffd700 1 -- 172.21.15.67:0/2303757565 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3634021070 con 0x7f363c108580 2024-05-03T17:19:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.086+0000 7f3632ffd700 1 -- 172.21.15.67:0/2303757565 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3634007b80 con 0x7f363c108580 2024-05-03T17:19:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.086+0000 7f3643a8c700 1 -- 172.21.15.67:0/2303757565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f363c1a81e0 con 0x7f363c108580 2024-05-03T17:19:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.086+0000 7f3632ffd700 1 -- 172.21.15.67:0/2303757565 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3634005040 con 0x7f363c108580 2024-05-03T17:19:36.089 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.086+0000 7f3643a8c700 1 -- 172.21.15.67:0/2303757565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f363c1a8690 con 0x7f363c108580 2024-05-03T17:19:36.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.087+0000 7f3643a8c700 1 -- 172.21.15.67:0/2303757565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f363c06a190 con 0x7f363c108580 2024-05-03T17:19:36.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.088+0000 7f3632ffd700 1 -- 172.21.15.67:0/2303757565 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f3634024070 con 0x7f363c108580 2024-05-03T17:19:36.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.088+0000 7f3632ffd700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f36280b2880 0x7f36280b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.089+0000 7f3641828700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f36280b2880 0x7f36280b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.089+0000 7f3632ffd700 1 -- 172.21.15.67:0/2303757565 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f363400f020 con 0x7f363c108580 2024-05-03T17:19:36.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.089+0000 7f3641828700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f36280b2880 0x7f36280b4d40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f363800f410 tx=0x7f363800a430 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:36.094 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.093+0000 7f3632ffd700 1 -- 172.21.15.67:0/2303757565 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f363402b020 con 0x7f363c108580 2024-05-03T17:19:36.312 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.311+0000 7f3643a8c700 1 -- 172.21.15.67:0/2303757565 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f363c1a89d0 con 0x7f363c108580 2024-05-03T17:19:36.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.313+0000 7f3632ffd700 1 -- 172.21.15.67:0/2303757565 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18373 (secure 0 0 0) 0x7f36340ab4c0 con 0x7f363c108580 2024-05-03T17:19:36.314 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:19:36.314 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":72,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","created":"2024-05-03T17:12:11.709381+0000","modified":"2024-05-03T17:19:30.317916+0000","last_up_change":"2024-05-03T17:19:29.310797+0000","last_in_change":"2024-05-03T17:19:08.848654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T17:15:36.413266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80962eb1-a506-49bb-a528-dee035eb4322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6803","nonce":1754365741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6805","nonce":1754365741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6809","nonce":1754365741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6807","nonce":1754365741}]},"public_addr":"172.21.15.67:6803/1754365741","cluster_addr":"172.21.15.67:6805/1754365741","heartbeat_back_addr":"172.21.15.67:6809/1754365741","heartbeat_front_addr":"172.21.15.67:6807/1754365741","state":["exists","up"]},{"osd":1,"uuid":"d7721d5f-4373-4cb2-bb57-bc368d6bb818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6801","nonce":1006339176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6803","nonce":1006339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6807","nonce":1006339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6805","nonce":1006339176}]},"public_addr":"172.21.15.70:6801/1006339176","cluster_addr":"172.21.15.70:6803/1006339176","heartbeat_back_addr":"172.21.15.70:6807/1006339176","heartbeat_front_addr":"172.21.15.70:6805/1006339176","state":["exists","up"]},{"osd":2,"uuid":"9992fd98-821b-40ce-b051-89befbfd87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6801","nonce":1888803264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6803","nonce":1888803264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6807","nonce":1888803264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6805","nonce":1888803264}]},"public_addr":"172.21.15.82:6801/1888803264","cluster_addr":"172.21.15.82:6803/1888803264","heartbeat_back_addr":"172.21.15.82:6807/1888803264","heartbeat_front_addr":"172.21.15.82:6805/1888803264","state":["exists","up"]},{"osd":3,"uuid":"14bfe6a6-0bc8-4766-8d36-fbaecc61a396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6811","nonce":313717723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6813","nonce":313717723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6817","nonce":313717723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6815","nonce":313717723}]},"public_addr":"172.21.15.67:6811/313717723","cluster_addr":"172.21.15.67:6813/313717723","heartbeat_back_addr":"172.21.15.67:6817/313717723","heartbeat_front_addr":"172.21.15.67:6815/313717723","state":["exists","up"]},{"osd":4,"uuid":"5a44aeb8-aabf-4824-be65-2c7170dbee52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6809","nonce":3526300828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6811","nonce":3526300828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6815","nonce":3526300828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6813","nonce":3526300828}]},"public_addr":"172.21.15.70:6809/3526300828","cluster_addr":"172.21.15.70:6811/3526300828","heartbeat_back_addr":"172.21.15.70:6815/3526300828","heartbeat_front_addr":"172.21.15.70:6813/3526300828","state":["exists","up"]},{"osd":5,"uuid":"a6ac56df-466e-4096-8fbf-835f5b544297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6809","nonce":3754223290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6811","nonce":3754223290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6815","nonce":3754223290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6813","nonce":3754223290}]},"public_addr":"172.21.15.82:6809/3754223290","cluster_addr":"172.21.15.82:6811/3754223290","heartbeat_back_addr":"172.21.15.82:6815/3754223290","heartbeat_front_addr":"172.21.15.82:6813/3754223290","state":["exists","up"]},{"osd":6,"uuid":"4d8a5e8a-66ac-462e-b960-24c0ee248fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6819","nonce":4033769161}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6821","nonce":4033769161}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6825","nonce":4033769161}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6823","nonce":4033769161}]},"public_addr":"172.21.15.67:6819/4033769161","cluster_addr":"172.21.15.67:6821/4033769161","heartbeat_back_addr":"172.21.15.67:6825/4033769161","heartbeat_front_addr":"172.21.15.67:6823/4033769161","state":["exists","up"]},{"osd":7,"uuid":"eb983837-8172-4802-9843-6360cda8f4be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6817","nonce":3366146173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6819","nonce":3366146173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6823","nonce":3366146173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6821","nonce":3366146173}]},"public_addr":"172.21.15.70:6817/3366146173","cluster_addr":"172.21.15.70:6819/3366146173","heartbeat_back_addr":"172.21.15.70:6823/3366146173","heartbeat_front_addr":"172.21.15.70:6821/3366146173","state":["exists","up"]},{"osd":8,"uuid":"c9d4ce56-e940-48d1-a9b5-437139e3fed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6817","nonce":347699286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6819","nonce":347699286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6823","nonce":347699286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6821","nonce":347699286}]},"public_addr":"172.21.15.82:6817/347699286","cluster_addr":"172.21.15.82:6819/347699286","heartbeat_back_addr":"172.21.15.82:6823/347699286","heartbeat_front_addr":"172.21.15.82:6821/347699286","state":["exists","up"]},{"osd":9,"uuid":"09596d19-f2d7-485e-a028-9ce5468c6651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6827","nonce":2600095198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6829","nonce":2600095198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6833","nonce":2600095198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6831","nonce":2600095198}]},"public_addr":"172.21.15.67:6827/2600095198","cluster_addr":"172.21.15.67:6829/2600095198","heartbeat_back_addr":"172.21.15.67:6833/2600095198","heartbeat_front_addr":"172.21.15.67:6831/2600095198","state":["exists","up"]},{"osd":10,"uuid":"ec4cffbc-0127-47e5-8176-0972f74a5514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6825","nonce":2897583629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6827","nonce":2897583629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6831","nonce":2897583629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6829","nonce":2897583629}]},"public_addr":"172.21.15.70:6825/2897583629","cluster_addr":"172.21.15.70:6827/2897583629","heartbeat_back_addr":"172.21.15.70:6831/2897583629","heartbeat_front_addr":"172.21.15.70:6829/2897583629","state":["exists","up"]},{"osd":11,"uuid":"a8acfb0b-98e8-49e1-a114-f33aa8160b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6825","nonce":4082656149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6827","nonce":4082656149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6831","nonce":4082656149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6829","nonce":4082656149}]},"public_addr":"172.21.15.82:6825/4082656149","cluster_addr":"172.21.15.82:6827/4082656149","heartbeat_back_addr":"172.21.15.82:6831/4082656149","heartbeat_front_addr":"172.21.15.82:6829/4082656149","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:14:45.596665+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:09.538408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:33.608945+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:59.803102+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:25.187966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:49.892160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:16.575973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:42.115955+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:07.655558+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:35.356779+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:01.144695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:27.577513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/2131289004":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/3335149382":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/992151181":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2614686067":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/3137329076":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2323686292":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/2579128777":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/3335149382":"2024-05-04T17:13:06.324002+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T17:19:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.315+0000 7f3643a8c700 1 -- 172.21.15.67:0/2303757565 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f36280b2880 msgr2=0x7f36280b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.315+0000 7f3643a8c700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f36280b2880 0x7f36280b4d40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f363800f410 tx=0x7f363800a430 comp rx=0 tx=0).stop 2024-05-03T17:19:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.315+0000 7f3643a8c700 1 -- 172.21.15.67:0/2303757565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f363c108580 msgr2=0x7f363c103ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.315+0000 7f3643a8c700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f363c108580 0x7f363c103ac0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f363400b070 tx=0x7f3634004ce0 comp rx=0 tx=0).stop 2024-05-03T17:19:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.315+0000 7f3643a8c700 1 -- 172.21.15.67:0/2303757565 shutdown_connections 2024-05-03T17:19:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.315+0000 7f3643a8c700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f36280b2880 0x7f36280b4d40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.315+0000 7f3643a8c700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f363c108580 0x7f363c103ac0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:36.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.315+0000 7f3643a8c700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f363c101e60 0x7f363c103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:36.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.315+0000 7f3643a8c700 1 --2- 172.21.15.67:0/2303757565 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f363c102820 0x7f363c103580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:36.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.315+0000 7f3643a8c700 1 -- 172.21.15.67:0/2303757565 >> 172.21.15.67:0/2303757565 conn(0x7f363c0776c0 msgr2=0x7f363c1008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:36.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.316+0000 7f3643a8c700 1 -- 172.21.15.67:0/2303757565 shutdown_connections 2024-05-03T17:19:36.317 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:36.316+0000 7f3643a8c700 1 -- 172.21.15.67:0/2303757565 wait complete. 2024-05-03T17:19:36.627 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:36 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2303757565' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:19:36.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:36 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2303757565' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:19:36.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:36 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2303757565' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:19:37.050 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-03T17:15:36.413266+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-03T17:19:37.051 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-03T17:19:37.297 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:19:37.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:37 smithi067 ceph-mon[29731]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:37.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:37 smithi082 ceph-mon[33535]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:37.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:37 smithi070 ceph-mon[33250]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.176+0000 7fd05f7b0700 1 -- 172.21.15.67:0/4278782660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd058109590 msgr2=0x7fd058109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.176+0000 7fd05f7b0700 1 --2- 172.21.15.67:0/4278782660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd058109590 0x7fd058109970 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd0540099c0 tx=0x7fd054009cd0 comp rx=0 tx=0).stop 2024-05-03T17:19:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.176+0000 7fd05f7b0700 1 -- 172.21.15.67:0/4278782660 shutdown_connections 2024-05-03T17:19:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.176+0000 7fd05f7b0700 1 --2- 172.21.15.67:0/4278782660 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd05806c630 0x7fd058110360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.176+0000 7fd05f7b0700 1 --2- 172.21.15.67:0/4278782660 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd058109590 0x7fd058109970 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.176+0000 7fd05f7b0700 1 --2- 172.21.15.67:0/4278782660 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd05806bc70 0x7fd05806c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:39.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.176+0000 7fd05f7b0700 1 -- 172.21.15.67:0/4278782660 >> 172.21.15.67:0/4278782660 conn(0x7fd058079440 msgr2=0x7fd058079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.176+0000 7fd05f7b0700 1 -- 172.21.15.67:0/4278782660 shutdown_connections 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.176+0000 7fd05f7b0700 1 -- 172.21.15.67:0/4278782660 wait complete. 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05f7b0700 1 Processor -- start 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05f7b0700 1 -- start start 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05f7b0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05806bc70 0x7fd058102410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05f7b0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd05806c630 0x7fd058102950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05f7b0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd058109590 0x7fd058102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05f7b0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd058113690 con 0x7fd05806bc70 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05f7b0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd058113510 con 0x7fd058109590 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05f7b0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd058113810 con 0x7fd05806c630 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05d54c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05806bc70 0x7fd058102410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05d54c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05806bc70 0x7fd058102410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36232/0 (socket says 172.21.15.67:36232) 2024-05-03T17:19:39.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05d54c700 1 -- 172.21.15.67:0/788080834 learned_addr learned my addr 172.21.15.67:0/788080834 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05cd4b700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd05806c630 0x7fd058102950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05dd4d700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd058109590 0x7fd058102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05d54c700 1 -- 172.21.15.67:0/788080834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd05806c630 msgr2=0x7fd058102950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05d54c700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd05806c630 0x7fd058102950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05d54c700 1 -- 172.21.15.67:0/788080834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd058109590 msgr2=0x7fd058102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05d54c700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd058109590 0x7fd058102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05d54c700 1 -- 172.21.15.67:0/788080834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0581a8070 con 0x7fd05806bc70 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05dd4d700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd058109590 0x7fd058102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.177+0000 7fd05d54c700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05806bc70 0x7fd058102410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd05400b580 tx=0x7fd0540047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.178+0000 7fd04e7fc700 1 -- 172.21.15.67:0/788080834 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054027070 con 0x7fd05806bc70 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.178+0000 7fd04e7fc700 1 -- 172.21.15.67:0/788080834 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd054003f20 con 0x7fd05806bc70 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.178+0000 7fd05f7b0700 1 -- 172.21.15.67:0/788080834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0581a8300 con 0x7fd05806bc70 2024-05-03T17:19:39.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.178+0000 7fd04e7fc700 1 -- 172.21.15.67:0/788080834 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd054022d70 con 0x7fd05806bc70 2024-05-03T17:19:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.178+0000 7fd05f7b0700 1 -- 172.21.15.67:0/788080834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0581a8750 con 0x7fd05806bc70 2024-05-03T17:19:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.179+0000 7fd05f7b0700 1 -- 172.21.15.67:0/788080834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd05810ba20 con 0x7fd05806bc70 2024-05-03T17:19:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.180+0000 7fd04e7fc700 1 -- 172.21.15.67:0/788080834 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fd054022500 con 0x7fd05806bc70 2024-05-03T17:19:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.180+0000 7fd04e7fc700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0440b2880 0x7fd0440b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.181+0000 7fd05cd4b700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0440b2880 0x7fd0440b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.181+0000 7fd04e7fc700 1 -- 172.21.15.67:0/788080834 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fd0540e2e30 con 0x7fd05806bc70 2024-05-03T17:19:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.181+0000 7fd05cd4b700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0440b2880 0x7fd0440b4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd058079a20 tx=0x7fd04800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:39.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.184+0000 7fd04e7fc700 1 -- 172.21.15.67:0/788080834 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fd054013070 con 0x7fd05806bc70 2024-05-03T17:19:39.399 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.399+0000 7fd05f7b0700 1 -- 172.21.15.67:0/788080834 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd05810b400 con 0x7fd05806bc70 2024-05-03T17:19:39.400 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.399+0000 7fd04e7fc700 1 -- 172.21.15.67:0/788080834 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7fd0540ab080 con 0x7fd05806bc70 2024-05-03T17:19:39.400 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2024-05-03T17:19:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.401+0000 7fd05f7b0700 1 -- 172.21.15.67:0/788080834 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0440b2880 msgr2=0x7fd0440b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.401+0000 7fd05f7b0700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0440b2880 0x7fd0440b4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd058079a20 tx=0x7fd04800b3c0 comp rx=0 tx=0).stop 2024-05-03T17:19:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.401+0000 7fd05f7b0700 1 -- 172.21.15.67:0/788080834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05806bc70 msgr2=0x7fd058102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.401+0000 7fd05f7b0700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05806bc70 0x7fd058102410 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd05400b580 tx=0x7fd0540047b0 comp rx=0 tx=0).stop 2024-05-03T17:19:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.401+0000 7fd05f7b0700 1 -- 172.21.15.67:0/788080834 shutdown_connections 2024-05-03T17:19:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.401+0000 7fd05f7b0700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0440b2880 0x7fd0440b4d40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.401+0000 7fd05f7b0700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd058109590 0x7fd058102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:39.402 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.401+0000 7fd05f7b0700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd05806bc70 0x7fd058102410 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.401+0000 7fd05f7b0700 1 --2- 172.21.15.67:0/788080834 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd05806c630 0x7fd058102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.401+0000 7fd05f7b0700 1 -- 172.21.15.67:0/788080834 >> 172.21.15.67:0/788080834 conn(0x7fd058079440 msgr2=0x7fd058100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.401+0000 7fd05f7b0700 1 -- 172.21.15.67:0/788080834 shutdown_connections 2024-05-03T17:19:39.403 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:39.401+0000 7fd05f7b0700 1 -- 172.21.15.67:0/788080834 wait complete. 2024-05-03T17:19:39.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:39 smithi082 ceph-mon[33535]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:39.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:39 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/788080834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-03T17:19:39.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:39 smithi070 ceph-mon[33250]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:39.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:39 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/788080834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-03T17:19:39.815 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:39 smithi067 ceph-mon[29731]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:39.815 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:39 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/788080834' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-03T17:19:40.202 INFO:tasks.cephadm:Adding mds.a on smithi067 2024-05-03T17:19:40.202 INFO:tasks.cephadm:Adding mds.d on smithi067 2024-05-03T17:19:40.202 INFO:tasks.cephadm:Adding mds.g on smithi067 2024-05-03T17:19:40.202 INFO:tasks.cephadm:Adding mds.j on smithi067 2024-05-03T17:19:40.202 INFO:tasks.cephadm:Adding mds.b on smithi070 2024-05-03T17:19:40.202 INFO:tasks.cephadm:Adding mds.e on smithi070 2024-05-03T17:19:40.202 INFO:tasks.cephadm:Adding mds.h on smithi070 2024-05-03T17:19:40.202 INFO:tasks.cephadm:Adding mds.k on smithi070 2024-05-03T17:19:40.202 INFO:tasks.cephadm:Adding mds.c on smithi082 2024-05-03T17:19:40.203 INFO:tasks.cephadm:Adding mds.f on smithi082 2024-05-03T17:19:40.203 INFO:tasks.cephadm:Adding mds.i on smithi082 2024-05-03T17:19:40.203 INFO:tasks.cephadm:Adding mds.l on smithi082 2024-05-03T17:19:40.203 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph orch apply mds all '12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi082=c;smithi082=f;smithi082=i;smithi082=l' 2024-05-03T17:19:40.451 INFO:teuthology.orchestra.run.smithi082.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.c/config 2024-05-03T17:19:41.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:41 smithi082 ceph-mon[33535]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:41.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:41 smithi070 ceph-mon[33250]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:41.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:41 smithi067 ceph-mon[29731]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:42.211 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.209+0000 7f013375e700 1 -- 172.21.15.82:0/2914708408 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f012c1023f0 msgr2=0x7f012c102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.209+0000 7f013375e700 1 --2- 172.21.15.82:0/2914708408 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f012c1023f0 0x7f012c102810 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f012800b0a0 tx=0x7f012800b3b0 comp rx=0 tx=0).stop 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.209+0000 7f013375e700 1 -- 172.21.15.82:0/2914708408 shutdown_connections 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.209+0000 7f013375e700 1 --2- 172.21.15.82:0/2914708408 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f012c103f90 0x7f012c10a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.209+0000 7f013375e700 1 --2- 172.21.15.82:0/2914708408 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f012c1023f0 0x7f012c102810 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.209+0000 7f013375e700 1 --2- 172.21.15.82:0/2914708408 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f012c1035f0 0x7f012c103a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.209+0000 7f013375e700 1 -- 172.21.15.82:0/2914708408 >> 172.21.15.82:0/2914708408 conn(0x7f012c0fd990 msgr2=0x7f012c0ffdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.209+0000 7f013375e700 1 -- 172.21.15.82:0/2914708408 shutdown_connections 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.209+0000 7f013375e700 1 -- 172.21.15.82:0/2914708408 wait complete. 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f013375e700 1 Processor -- start 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f013375e700 1 -- start start 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f013375e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f012c1023f0 0x7f012c1996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f013375e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f012c1035f0 0x7f012c199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:42.212 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f013375e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f012c103f90 0x7f012c1a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f013375e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f012c10dee0 con 0x7f012c1035f0 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f013375e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f012c10dd60 con 0x7f012c1023f0 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f013375e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f012c10e060 con 0x7f012c103f90 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f0131cfb700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f012c103f90 0x7f012c1a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f0131cfb700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f012c103f90 0x7f012c1a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.82:59972/0 (socket says 172.21.15.82:59972) 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f0131cfb700 1 -- 172.21.15.82:0/376580898 learned_addr learned my addr 172.21.15.82:0/376580898 (peer_addr_for_me v2:172.21.15.82:0/0) 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f01314fa700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f012c1023f0 0x7f012c1996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.210+0000 7f0130cf9700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f012c1035f0 0x7f012c199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.211+0000 7f0131cfb700 1 -- 172.21.15.82:0/376580898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f012c1023f0 msgr2=0x7f012c1996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.211+0000 7f0131cfb700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f012c1023f0 0x7f012c1996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.211+0000 7f0131cfb700 1 -- 172.21.15.82:0/376580898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f012c1035f0 msgr2=0x7f012c199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.211+0000 7f0131cfb700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f012c1035f0 0x7f012c199c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.211+0000 7f0131cfb700 1 -- 172.21.15.82:0/376580898 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012c1a1320 con 0x7f012c103f90 2024-05-03T17:19:42.213 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.211+0000 7f0130cf9700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f012c1035f0 0x7f012c199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:19:42.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.211+0000 7f0131cfb700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f012c103f90 0x7f012c1a0c80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f012400b580 tx=0x7f012400b890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:42.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.211+0000 7f01227fc700 1 -- 172.21.15.82:0/376580898 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124015020 con 0x7f012c103f90 2024-05-03T17:19:42.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.211+0000 7f013375e700 1 -- 172.21.15.82:0/376580898 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f012c1a1610 con 0x7f012c103f90 2024-05-03T17:19:42.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.211+0000 7f01227fc700 1 -- 172.21.15.82:0/376580898 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0124011ce0 con 0x7f012c103f90 2024-05-03T17:19:42.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.211+0000 7f01227fc700 1 -- 172.21.15.82:0/376580898 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0124010dd0 con 0x7f012c103f90 2024-05-03T17:19:42.214 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.211+0000 7f013375e700 1 -- 172.21.15.82:0/376580898 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f012c1a1bd0 con 0x7f012c103f90 2024-05-03T17:19:42.215 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.212+0000 7f013375e700 1 -- 172.21.15.82:0/376580898 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f012c06a190 con 0x7f012c103f90 2024-05-03T17:19:42.215 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.213+0000 7f01227fc700 1 -- 172.21.15.82:0/376580898 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f0124018020 con 0x7f012c103f90 2024-05-03T17:19:42.215 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.214+0000 7f01227fc700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f01180b2880 0x7f01180b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:42.215 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.214+0000 7f01227fc700 1 -- 172.21.15.82:0/376580898 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f01240e1a80 con 0x7f012c103f90 2024-05-03T17:19:42.215 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.214+0000 7f01314fa700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f01180b2880 0x7f01180b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:42.215 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.215+0000 7f01314fa700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f01180b2880 0x7f01180b4d40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0128009fd0 tx=0x7f0128006000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:42.222 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.222+0000 7f01227fc700 1 -- 172.21.15.82:0/376580898 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f012401f020 con 0x7f012c103f90 2024-05-03T17:19:42.444 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.444+0000 7f013375e700 1 -- 172.21.15.82:0/376580898 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi082=c;smithi082=f;smithi082=i;smithi082=l", "target": ["mon-mgr", ""]}) -- 0x7f012c0ffbe0 con 0x7f01180b2880 2024-05-03T17:19:42.453 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.452+0000 7f01227fc700 1 -- 172.21.15.82:0/376580898 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f012c0ffbe0 con 0x7f01180b2880 2024-05-03T17:19:42.453 INFO:teuthology.orchestra.run.smithi082.stdout:Scheduled mds.all update... 2024-05-03T17:19:42.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.454+0000 7f013375e700 1 -- 172.21.15.82:0/376580898 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f01180b2880 msgr2=0x7f01180b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:42.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.454+0000 7f013375e700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f01180b2880 0x7f01180b4d40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f0128009fd0 tx=0x7f0128006000 comp rx=0 tx=0).stop 2024-05-03T17:19:42.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.454+0000 7f013375e700 1 -- 172.21.15.82:0/376580898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f012c103f90 msgr2=0x7f012c1a0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:42.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.454+0000 7f013375e700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f012c103f90 0x7f012c1a0c80 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f012400b580 tx=0x7f012400b890 comp rx=0 tx=0).stop 2024-05-03T17:19:42.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.455+0000 7f013375e700 1 -- 172.21.15.82:0/376580898 shutdown_connections 2024-05-03T17:19:42.455 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.455+0000 7f013375e700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f01180b2880 0x7f01180b4d40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:42.456 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.455+0000 7f013375e700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f012c103f90 0x7f012c1a0c80 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:42.456 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.455+0000 7f013375e700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f012c1023f0 0x7f012c1996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:42.456 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.455+0000 7f013375e700 1 --2- 172.21.15.82:0/376580898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f012c1035f0 0x7f012c199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:42.456 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.455+0000 7f013375e700 1 -- 172.21.15.82:0/376580898 >> 172.21.15.82:0/376580898 conn(0x7f012c0fd990 msgr2=0x7f012c0ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:42.456 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.455+0000 7f013375e700 1 -- 172.21.15.82:0/376580898 shutdown_connections 2024-05-03T17:19:42.456 INFO:teuthology.orchestra.run.smithi082.stderr:2024-05-03T17:19:42.455+0000 7f013375e700 1 -- 172.21.15.82:0/376580898 wait complete. 2024-05-03T17:19:43.211 DEBUG:teuthology.orchestra.run.smithi067:mds.a> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.a.service 2024-05-03T17:19:43.214 DEBUG:teuthology.orchestra.run.smithi067:mds.d> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.d.service 2024-05-03T17:19:43.257 DEBUG:teuthology.orchestra.run.smithi067:mds.g> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.g.service 2024-05-03T17:19:43.299 DEBUG:teuthology.orchestra.run.smithi067:mds.j> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.j.service 2024-05-03T17:19:43.342 DEBUG:teuthology.orchestra.run.smithi070:mds.b> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.b.service 2024-05-03T17:19:43.345 DEBUG:teuthology.orchestra.run.smithi070:mds.e> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.e.service 2024-05-03T17:19:43.388 DEBUG:teuthology.orchestra.run.smithi070:mds.h> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.h.service 2024-05-03T17:19:43.431 DEBUG:teuthology.orchestra.run.smithi070:mds.k> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.k.service 2024-05-03T17:19:43.474 DEBUG:teuthology.orchestra.run.smithi082:mds.c> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.c.service 2024-05-03T17:19:43.477 DEBUG:teuthology.orchestra.run.smithi082:mds.f> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.f.service 2024-05-03T17:19:43.522 DEBUG:teuthology.orchestra.run.smithi082:mds.i> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.i.service 2024-05-03T17:19:43.565 DEBUG:teuthology.orchestra.run.smithi082:mds.l> sudo journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.l.service 2024-05-03T17:19:43.608 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-03T17:19:43.608 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:19:43.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:43 smithi067 ceph-mon[29731]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:43.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:43 smithi067 ceph-mon[29731]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi082=c;smithi082=f;smithi082=i;smithi082=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:19:43.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:43 smithi067 ceph-mon[29731]: Saving service mds.all spec with placement smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi082=c;smithi082=f;smithi082=i;smithi082=l;count:12 2024-05-03T17:19:43.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:43.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:19:43.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:43.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:43.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:43.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:43.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:43.631 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:43.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:43 smithi082 ceph-mon[33535]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:43 smithi082 ceph-mon[33535]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi082=c;smithi082=f;smithi082=i;smithi082=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:19:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:43 smithi082 ceph-mon[33535]: Saving service mds.all spec with placement smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi082=c;smithi082=f;smithi082=i;smithi082=l;count:12 2024-05-03T17:19:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:19:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:43.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:43 smithi070 ceph-mon[33250]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:43 smithi070 ceph-mon[33250]: from='client.24385 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi082=c;smithi082=f;smithi082=i;smithi082=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:19:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:43 smithi070 ceph-mon[33250]: Saving service mds.all spec with placement smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi082=c;smithi082=f;smithi082=i;smithi082=l;count:12 2024-05-03T17:19:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:19:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:19:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:43.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:43.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:43.809 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:43.852 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:19:44.702 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:44 smithi082 ceph-mon[33535]: Deploying daemon mds.l on smithi082 2024-05-03T17:19:44.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:44 smithi067 ceph-mon[29731]: Deploying daemon mds.l on smithi082 2024-05-03T17:19:44.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:44 smithi070 ceph-mon[33250]: Deploying daemon mds.l on smithi082 2024-05-03T17:19:45.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:45 smithi082 ceph-mon[33535]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:45.568 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:45 smithi067 ceph-mon[29731]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.678+0000 7fca54f60700 1 -- 172.21.15.67:0/169291805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5006b180 msgr2=0x7fca5006b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.678+0000 7fca54f60700 1 --2- 172.21.15.67:0/169291805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5006b180 0x7fca5006b600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fca38009990 tx=0x7fca38009ca0 comp rx=0 tx=0).stop 2024-05-03T17:19:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.678+0000 7fca54f60700 1 -- 172.21.15.67:0/169291805 shutdown_connections 2024-05-03T17:19:45.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.678+0000 7fca54f60700 1 --2- 172.21.15.67:0/169291805 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca5006bb40 0x7fca50110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.678+0000 7fca54f60700 1 --2- 172.21.15.67:0/169291805 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca50109510 0x7fca501098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.678+0000 7fca54f60700 1 --2- 172.21.15.67:0/169291805 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5006b180 0x7fca5006b600 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.678+0000 7fca54f60700 1 -- 172.21.15.67:0/169291805 >> 172.21.15.67:0/169291805 conn(0x7fca50079350 msgr2=0x7fca50079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.678+0000 7fca54f60700 1 -- 172.21.15.67:0/169291805 shutdown_connections 2024-05-03T17:19:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.678+0000 7fca54f60700 1 -- 172.21.15.67:0/169291805 wait complete. 2024-05-03T17:19:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.678+0000 7fca54f60700 1 Processor -- start 2024-05-03T17:19:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.678+0000 7fca54f60700 1 -- start start 2024-05-03T17:19:45.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca54f60700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca5006b180 0x7fca5019bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:45.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca54f60700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5006bb40 0x7fca5019c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:45.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca54f60700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca50109510 0x7fca501a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:45.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca54f60700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fca501136a0 con 0x7fca5006bb40 2024-05-03T17:19:45.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca54f60700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fca50113520 con 0x7fca5006b180 2024-05-03T17:19:45.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca54f60700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fca50113820 con 0x7fca50109510 2024-05-03T17:19:45.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca47fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5006bb40 0x7fca5019c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:45.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca47fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5006bb40 0x7fca5019c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36254/0 (socket says 172.21.15.67:36254) 2024-05-03T17:19:45.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca47fff700 1 -- 172.21.15.67:0/753322436 learned_addr learned my addr 172.21.15.67:0/753322436 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:19:45.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca4e59c700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca5006b180 0x7fca5019bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca4ed9d700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca50109510 0x7fca501a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca47fff700 1 -- 172.21.15.67:0/753322436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca50109510 msgr2=0x7fca501a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca47fff700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca50109510 0x7fca501a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca47fff700 1 -- 172.21.15.67:0/753322436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca5006b180 msgr2=0x7fca5019bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca47fff700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca5006b180 0x7fca5019bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca47fff700 1 -- 172.21.15.67:0/753322436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca501a0e00 con 0x7fca5006bb40 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca47fff700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5006bb40 0x7fca5019c330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fca38000c00 tx=0x7fca380047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.679+0000 7fca477fe700 1 -- 172.21.15.67:0/753322436 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38027070 con 0x7fca5006bb40 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.680+0000 7fca477fe700 1 -- 172.21.15.67:0/753322436 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fca38003f20 con 0x7fca5006bb40 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.680+0000 7fca54f60700 1 -- 172.21.15.67:0/753322436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca501a1090 con 0x7fca5006bb40 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.680+0000 7fca477fe700 1 -- 172.21.15.67:0/753322436 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca38022ca0 con 0x7fca5006bb40 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.680+0000 7fca54f60700 1 -- 172.21.15.67:0/753322436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca501a8970 con 0x7fca5006bb40 2024-05-03T17:19:45.684 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.681+0000 7fca54f60700 1 -- 172.21.15.67:0/753322436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca50104bd0 con 0x7fca5006bb40 2024-05-03T17:19:45.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.683+0000 7fca477fe700 1 -- 172.21.15.67:0/753322436 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fca3802a070 con 0x7fca5006bb40 2024-05-03T17:19:45.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.683+0000 7fca477fe700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fca3c0b2880 0x7fca3c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:45.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.684+0000 7fca4e59c700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fca3c0b2880 0x7fca3c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:45.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.684+0000 7fca477fe700 1 -- 172.21.15.67:0/753322436 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fca380e3070 con 0x7fca5006bb40 2024-05-03T17:19:45.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.684+0000 7fca4e59c700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fca3c0b2880 0x7fca3c0b4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fca40009a10 tx=0x7fca40009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:45.688 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.687+0000 7fca477fe700 1 -- 172.21.15.67:0/753322436 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fca38031070 con 0x7fca5006bb40 2024-05-03T17:19:45.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:45 smithi070 ceph-mon[33250]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:45.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.949+0000 7fca54f60700 1 -- 172.21.15.67:0/753322436 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fca5006a190 con 0x7fca5006bb40 2024-05-03T17:19:45.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.949+0000 7fca477fe700 1 -- 172.21.15.67:0/753322436 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7fca5006a190 con 0x7fca5006bb40 2024-05-03T17:19:45.950 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:19:45.950 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-03T17:19:45.952 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.952+0000 7fca54f60700 1 -- 172.21.15.67:0/753322436 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fca3c0b2880 msgr2=0x7fca3c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:45.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.952+0000 7fca54f60700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fca3c0b2880 0x7fca3c0b4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fca40009a10 tx=0x7fca40009190 comp rx=0 tx=0).stop 2024-05-03T17:19:45.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.952+0000 7fca54f60700 1 -- 172.21.15.67:0/753322436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5006bb40 msgr2=0x7fca5019c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:45.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.952+0000 7fca54f60700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5006bb40 0x7fca5019c330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fca38000c00 tx=0x7fca380047b0 comp rx=0 tx=0).stop 2024-05-03T17:19:45.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.952+0000 7fca54f60700 1 -- 172.21.15.67:0/753322436 shutdown_connections 2024-05-03T17:19:45.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.952+0000 7fca54f60700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fca3c0b2880 0x7fca3c0b4d40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:45.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.952+0000 7fca54f60700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fca50109510 0x7fca501a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:45.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.952+0000 7fca54f60700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fca5006b180 0x7fca5019bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:45.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.952+0000 7fca54f60700 1 --2- 172.21.15.67:0/753322436 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca5006bb40 0x7fca5019c330 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:45.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.952+0000 7fca54f60700 1 -- 172.21.15.67:0/753322436 >> 172.21.15.67:0/753322436 conn(0x7fca50079350 msgr2=0x7fca5010e680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:45.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.952+0000 7fca54f60700 1 -- 172.21.15.67:0/753322436 shutdown_connections 2024-05-03T17:19:45.953 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:45.952+0000 7fca54f60700 1 -- 172.21.15.67:0/753322436 wait complete. 2024-05-03T17:19:45.954 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 2 2024-05-03T17:19:46.264 INFO:journalctl@ceph.mds.l.smithi082.stdout:May 03 17:19:45 smithi082 systemd[1]: Started Ceph mds.l for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:19:47.036 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:46 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.82:6832/3927012974,v1:172.21.15.82:6833/3927012974] up:boot 2024-05-03T17:19:47.036 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:46 smithi070 ceph-mon[33250]: fsmap 1 up:standby 2024-05-03T17:19:47.036 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-03T17:19:47.036 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:46 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/753322436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:19:47.036 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:47.036 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:47.037 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:47.037 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:47.037 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:47.037 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:47.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:46 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.82:6832/3927012974,v1:172.21.15.82:6833/3927012974] up:boot 2024-05-03T17:19:47.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:46 smithi067 ceph-mon[29731]: fsmap 1 up:standby 2024-05-03T17:19:47.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-03T17:19:47.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:46 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/753322436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:19:47.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:47.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:47.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:47.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:47.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:47.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:47.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:46 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.82:6832/3927012974,v1:172.21.15.82:6833/3927012974] up:boot 2024-05-03T17:19:47.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:46 smithi082 ceph-mon[33535]: fsmap 1 up:standby 2024-05-03T17:19:47.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-03T17:19:47.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:46 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/753322436' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:19:47.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:47.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:47.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:47.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:47.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:47.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:48.043 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:47 smithi070 ceph-mon[33250]: Deploying daemon mds.k on smithi070 2024-05-03T17:19:48.044 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:47 smithi070 ceph-mon[33250]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:48.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:47 smithi067 ceph-mon[29731]: Deploying daemon mds.k on smithi070 2024-05-03T17:19:48.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:47 smithi067 ceph-mon[29731]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:48.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:47 smithi082 ceph-mon[33535]: Deploying daemon mds.k on smithi070 2024-05-03T17:19:48.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:47 smithi082 ceph-mon[33535]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:48.308 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:19:48 smithi070 systemd[1]: Starting Ceph mds.k for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:19:48.690 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:19:48 smithi070 podman[62885]: 2024-05-03 17:19:48.340675584 +0000 UTC m=+0.018337983 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:19:48.690 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:19:48 smithi070 podman[62885]: 2024-05-03 17:19:48.442727923 +0000 UTC m=+0.120390322 container create e4257f73457477f32efe8d248ddc571f77a1a6acba192684a1e361a88fc4315e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-k, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:19:48.690 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:19:48 smithi070 podman[62885]: 2024-05-03 17:19:48.687412683 +0000 UTC m=+0.365075094 container init e4257f73457477f32efe8d248ddc571f77a1a6acba192684a1e361a88fc4315e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-k, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-03T17:19:48.690 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:19:48 smithi070 podman[62885]: 2024-05-03 17:19:48.689890704 +0000 UTC m=+0.367553112 container start e4257f73457477f32efe8d248ddc571f77a1a6acba192684a1e361a88fc4315e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-k, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:19:48.755 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:19:49.027 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:19:49.058 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:19:48 smithi070 ceph-mds[62904]: -- 172.21.15.70:0/3034905326 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c7ae085a40 con 0x55c7ae059000 2024-05-03T17:19:49.058 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:19:48 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-k[62900]: starting mds.k at 2024-05-03T17:19:49.058 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:19:48 smithi070 bash[62885]: e4257f73457477f32efe8d248ddc571f77a1a6acba192684a1e361a88fc4315e 2024-05-03T17:19:49.058 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:19:48 smithi070 systemd[1]: Started Ceph mds.k for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:19:50.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:49 smithi067 ceph-mon[29731]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:50.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:50.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:50.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:50.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:50.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:50.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:50.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:49 smithi067 ceph-mon[29731]: Deploying daemon mds.j on smithi067 2024-05-03T17:19:50.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:49 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.70:6832/2034294041,v1:172.21.15.70:6833/2034294041] up:boot 2024-05-03T17:19:50.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:49 smithi067 ceph-mon[29731]: fsmap 2 up:standby 2024-05-03T17:19:50.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-03T17:19:50.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:49 smithi082 ceph-mon[33535]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:49 smithi082 ceph-mon[33535]: Deploying daemon mds.j on smithi067 2024-05-03T17:19:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:49 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.70:6832/2034294041,v1:172.21.15.70:6833/2034294041] up:boot 2024-05-03T17:19:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:49 smithi082 ceph-mon[33535]: fsmap 2 up:standby 2024-05-03T17:19:50.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-03T17:19:50.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:49 smithi070 ceph-mon[33250]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:50.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:50.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:50.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:50.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:50.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:50.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:50.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:49 smithi070 ceph-mon[33250]: Deploying daemon mds.j on smithi067 2024-05-03T17:19:50.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:49 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.70:6832/2034294041,v1:172.21.15.70:6833/2034294041] up:boot 2024-05-03T17:19:50.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:49 smithi070 ceph-mon[33250]: fsmap 2 up:standby 2024-05-03T17:19:50.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-03T17:19:52.145 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:51 smithi067 ceph-mon[29731]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.163+0000 7f97f1299700 1 -- 172.21.15.67:0/1318863289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec102d50 msgr2=0x7f97ec10f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.163+0000 7f97f1299700 1 --2- 172.21.15.67:0/1318863289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec102d50 0x7f97ec10f240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f97e400b0a0 tx=0x7f97e400b3b0 comp rx=0 tx=0).stop 2024-05-03T17:19:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.163+0000 7f97f1299700 1 -- 172.21.15.67:0/1318863289 shutdown_connections 2024-05-03T17:19:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.163+0000 7f97f1299700 1 --2- 172.21.15.67:0/1318863289 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec102d50 0x7f97ec10f240 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.163+0000 7f97f1299700 1 --2- 172.21.15.67:0/1318863289 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97ec108400 0x7f97ec1087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.163+0000 7f97f1299700 1 --2- 172.21.15.67:0/1318863289 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97ec102390 0x7f97ec102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.163+0000 7f97f1299700 1 -- 172.21.15.67:0/1318863289 >> 172.21.15.67:0/1318863289 conn(0x7f97ec079310 msgr2=0x7f97ec079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:52.165 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.163+0000 7f97f1299700 1 -- 172.21.15.67:0/1318863289 shutdown_connections 2024-05-03T17:19:52.166 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.163+0000 7f97f1299700 1 -- 172.21.15.67:0/1318863289 wait complete. 2024-05-03T17:19:52.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.164+0000 7f97f1299700 1 Processor -- start 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.164+0000 7f97f1299700 1 -- start start 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97f1299700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97ec102390 0x7f97ec19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97f1299700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec102d50 0x7f97ec19ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97f1299700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97ec108400 0x7f97ec1a5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97f1299700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec112550 con 0x7f97ec102d50 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97f1299700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec1123d0 con 0x7f97ec102390 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97f1299700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f97ec1126d0 con 0x7f97ec108400 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97ea7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec102d50 0x7f97ec19ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97ea7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec102d50 0x7f97ec19ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39084/0 (socket says 172.21.15.67:39084) 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97ea7fc700 1 -- 172.21.15.67:0/1798671566 learned_addr learned my addr 172.21.15.67:0/1798671566 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97eaffd700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97ec102390 0x7f97ec19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97eb7fe700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97ec108400 0x7f97ec1a5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97ea7fc700 1 -- 172.21.15.67:0/1798671566 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97ec108400 msgr2=0x7f97ec1a5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97ea7fc700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97ec108400 0x7f97ec1a5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97ea7fc700 1 -- 172.21.15.67:0/1798671566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97ec102390 msgr2=0x7f97ec19a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:52.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97ea7fc700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97ec102390 0x7f97ec19a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97ea7fc700 1 -- 172.21.15.67:0/1798671566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97ec1a5ed0 con 0x7f97ec102d50 2024-05-03T17:19:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97ea7fc700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec102d50 0x7f97ec19ae50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f97d400d980 tx=0x7f97d400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97e3fff700 1 -- 172.21.15.67:0/1798671566 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d4017070 con 0x7f97ec102d50 2024-05-03T17:19:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.165+0000 7f97f1299700 1 -- 172.21.15.67:0/1798671566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97ec1a6100 con 0x7f97ec102d50 2024-05-03T17:19:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.166+0000 7f97e3fff700 1 -- 172.21.15.67:0/1798671566 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97d4005020 con 0x7f97ec102d50 2024-05-03T17:19:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.166+0000 7f97f1299700 1 -- 172.21.15.67:0/1798671566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97ec1a6610 con 0x7f97ec102d50 2024-05-03T17:19:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.166+0000 7f97e3fff700 1 -- 172.21.15.67:0/1798671566 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d400fa10 con 0x7f97ec102d50 2024-05-03T17:19:52.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.166+0000 7f97f1299700 1 -- 172.21.15.67:0/1798671566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97ec103ab0 con 0x7f97ec102d50 2024-05-03T17:19:52.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.169+0000 7f97e3fff700 1 -- 172.21.15.67:0/1798671566 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f97d4004a30 con 0x7f97ec102d50 2024-05-03T17:19:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.170+0000 7f97e3fff700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f97d80b2900 0x7f97d80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.170+0000 7f97eaffd700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f97d80b2900 0x7f97d80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:52.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.170+0000 7f97e3fff700 1 -- 172.21.15.67:0/1798671566 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f97d40e1f80 con 0x7f97ec102d50 2024-05-03T17:19:52.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.170+0000 7f97eaffd700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f97d80b2900 0x7f97d80b4dc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009be0 tx=0x7f97dc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:52.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.173+0000 7f97e3fff700 1 -- 172.21.15.67:0/1798671566 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f97d4020020 con 0x7f97ec102d50 2024-05-03T17:19:52.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:51 smithi082 ceph-mon[33535]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:52.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:51 smithi070 ceph-mon[33250]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:52.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.443+0000 7f97f1299700 1 -- 172.21.15.67:0/1798671566 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f97ec02e510 con 0x7f97ec102d50 2024-05-03T17:19:52.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.444+0000 7f97e3fff700 1 -- 172.21.15.67:0/1798671566 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f97d40aa1c0 con 0x7f97ec102d50 2024-05-03T17:19:52.446 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:19:52.446 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-03T17:19:52.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.446+0000 7f97f1299700 1 -- 172.21.15.67:0/1798671566 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f97d80b2900 msgr2=0x7f97d80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:52.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.446+0000 7f97f1299700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f97d80b2900 0x7f97d80b4dc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f97dc009be0 tx=0x7f97dc009260 comp rx=0 tx=0).stop 2024-05-03T17:19:52.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.446+0000 7f97f1299700 1 -- 172.21.15.67:0/1798671566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec102d50 msgr2=0x7f97ec19ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:52.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.446+0000 7f97f1299700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec102d50 0x7f97ec19ae50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f97d400d980 tx=0x7f97d400dc90 comp rx=0 tx=0).stop 2024-05-03T17:19:52.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.446+0000 7f97f1299700 1 -- 172.21.15.67:0/1798671566 shutdown_connections 2024-05-03T17:19:52.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.446+0000 7f97f1299700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f97d80b2900 0x7f97d80b4dc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:52.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.446+0000 7f97f1299700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f97ec108400 0x7f97ec1a5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:52.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.446+0000 7f97f1299700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f97ec102390 0x7f97ec19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:52.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.446+0000 7f97f1299700 1 --2- 172.21.15.67:0/1798671566 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f97ec102d50 0x7f97ec19ae50 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:52.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.446+0000 7f97f1299700 1 -- 172.21.15.67:0/1798671566 >> 172.21.15.67:0/1798671566 conn(0x7f97ec079310 msgr2=0x7f97ec100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:52.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.446+0000 7f97f1299700 1 -- 172.21.15.67:0/1798671566 shutdown_connections 2024-05-03T17:19:52.449 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:52.446+0000 7f97f1299700 1 -- 172.21.15.67:0/1798671566 wait complete. 2024-05-03T17:19:52.449 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 3 2024-05-03T17:19:53.124 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:52 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1798671566' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:19:53.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:52 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1798671566' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:19:53.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:52 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1798671566' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:19:53.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:53 smithi067 ceph-mon[29731]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:53.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:53.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:53.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:53.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:53.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:53.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:53 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:54.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:53 smithi082 ceph-mon[33535]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:54.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:54.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:54.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:54.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:54.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:54.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:53 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:54.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:53 smithi070 ceph-mon[33250]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:54.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:54.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:54.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:54.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:54.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:54.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:53 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:55.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:19:55.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:54 smithi067 ceph-mon[29731]: Deploying daemon mds.d on smithi067 2024-05-03T17:19:55.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:54 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.67:6834/1814275265,v1:172.21.15.67:6835/1814275265] up:boot 2024-05-03T17:19:55.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:54 smithi067 ceph-mon[29731]: fsmap 3 up:standby 2024-05-03T17:19:55.181 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:54 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-03T17:19:55.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:54 smithi082 ceph-mon[33535]: Deploying daemon mds.d on smithi067 2024-05-03T17:19:55.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:54 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.67:6834/1814275265,v1:172.21.15.67:6835/1814275265] up:boot 2024-05-03T17:19:55.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:54 smithi082 ceph-mon[33535]: fsmap 3 up:standby 2024-05-03T17:19:55.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:54 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-03T17:19:55.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:54 smithi070 ceph-mon[33250]: Deploying daemon mds.d on smithi067 2024-05-03T17:19:55.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:54 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.67:6834/1814275265,v1:172.21.15.67:6835/1814275265] up:boot 2024-05-03T17:19:55.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:54 smithi070 ceph-mon[33250]: fsmap 3 up:standby 2024-05-03T17:19:55.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:54 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-03T17:19:55.736 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:19:56.141 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:55 smithi067 ceph-mon[29731]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:56.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:55 smithi082 ceph-mon[33535]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:56.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:55 smithi070 ceph-mon[33250]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.481+0000 7fe64c96e700 1 -- 172.21.15.67:0/1375588505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe648103750 msgr2=0x7fe648103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.481+0000 7fe64c96e700 1 --2- 172.21.15.67:0/1375588505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe648103750 0x7fe648103b30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe6380099c0 tx=0x7fe638009cd0 comp rx=0 tx=0).stop 2024-05-03T17:19:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.481+0000 7fe64c96e700 1 -- 172.21.15.67:0/1375588505 shutdown_connections 2024-05-03T17:19:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.481+0000 7fe64c96e700 1 --2- 172.21.15.67:0/1375588505 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe648108850 0x7fe6481103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:57.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.481+0000 7fe64c96e700 1 --2- 172.21.15.67:0/1375588505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe648103750 0x7fe648103b30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.481+0000 7fe64c96e700 1 --2- 172.21.15.67:0/1375588505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe648104100 0x7fe648108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.481+0000 7fe64c96e700 1 -- 172.21.15.67:0/1375588505 >> 172.21.15.67:0/1375588505 conn(0x7fe64807a600 msgr2=0x7fe64807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.481+0000 7fe64c96e700 1 -- 172.21.15.67:0/1375588505 shutdown_connections 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.481+0000 7fe64c96e700 1 -- 172.21.15.67:0/1375588505 wait complete. 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.482+0000 7fe64c96e700 1 Processor -- start 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.482+0000 7fe64c96e700 1 -- start start 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64c96e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe648103750 0x7fe64819bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64c96e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe648104100 0x7fe64819c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64c96e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe648108850 0x7fe6481a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64c96e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe6481136a0 con 0x7fe648103750 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64c96e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe648113520 con 0x7fe648104100 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64c96e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe648113820 con 0x7fe648108850 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe648103750 0x7fe64819bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe645d9b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe648104100 0x7fe64819c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64659c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe648103750 0x7fe64819bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55244/0 (socket says 172.21.15.67:55244) 2024-05-03T17:19:57.487 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64659c700 1 -- 172.21.15.67:0/2039754210 learned_addr learned my addr 172.21.15.67:0/2039754210 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe646d9d700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe648108850 0x7fe6481a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64659c700 1 -- 172.21.15.67:0/2039754210 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe648108850 msgr2=0x7fe6481a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64659c700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe648108850 0x7fe6481a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64659c700 1 -- 172.21.15.67:0/2039754210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe648104100 msgr2=0x7fe64819c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64659c700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe648104100 0x7fe64819c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64659c700 1 -- 172.21.15.67:0/2039754210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6481a0d70 con 0x7fe648103750 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.483+0000 7fe64659c700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe648103750 0x7fe64819bdb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe63800b580 tx=0x7fe6380047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.484+0000 7fe6377fe700 1 -- 172.21.15.67:0/2039754210 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe638027070 con 0x7fe648103750 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.484+0000 7fe64c96e700 1 -- 172.21.15.67:0/2039754210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6481a1000 con 0x7fe648103750 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.484+0000 7fe6377fe700 1 -- 172.21.15.67:0/2039754210 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe638003f20 con 0x7fe648103750 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.484+0000 7fe6377fe700 1 -- 172.21.15.67:0/2039754210 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe638022d10 con 0x7fe648103750 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.484+0000 7fe64c96e700 1 -- 172.21.15.67:0/2039754210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6481a88e0 con 0x7fe648103750 2024-05-03T17:19:57.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.485+0000 7fe64c96e700 1 -- 172.21.15.67:0/2039754210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe64806a190 con 0x7fe648103750 2024-05-03T17:19:57.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.487+0000 7fe6377fe700 1 -- 172.21.15.67:0/2039754210 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fe638022470 con 0x7fe648103750 2024-05-03T17:19:57.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.488+0000 7fe6377fe700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe6300b2880 0x7fe6300b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:19:57.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.488+0000 7fe6377fe700 1 -- 172.21.15.67:0/2039754210 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fe6380e2d90 con 0x7fe648103750 2024-05-03T17:19:57.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.488+0000 7fe645d9b700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe6300b2880 0x7fe6300b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:19:57.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.489+0000 7fe645d9b700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe6300b2880 0x7fe6300b4d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe648103250 tx=0x7fe63c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:19:57.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.491+0000 7fe6377fe700 1 -- 172.21.15.67:0/2039754210 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe638013070 con 0x7fe648103750 2024-05-03T17:19:57.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.765+0000 7fe64c96e700 1 -- 172.21.15.67:0/2039754210 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe6481a8ab0 con 0x7fe648103750 2024-05-03T17:19:57.767 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.766+0000 7fe6377fe700 1 -- 172.21.15.67:0/2039754210 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7fe6380aafe0 con 0x7fe648103750 2024-05-03T17:19:57.768 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:19:57.768 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-03T17:19:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.768+0000 7fe64c96e700 1 -- 172.21.15.67:0/2039754210 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe6300b2880 msgr2=0x7fe6300b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.768+0000 7fe64c96e700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe6300b2880 0x7fe6300b4d40 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe648103250 tx=0x7fe63c00b410 comp rx=0 tx=0).stop 2024-05-03T17:19:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.768+0000 7fe64c96e700 1 -- 172.21.15.67:0/2039754210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe648103750 msgr2=0x7fe64819bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:19:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.768+0000 7fe64c96e700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe648103750 0x7fe64819bdb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe63800b580 tx=0x7fe6380047b0 comp rx=0 tx=0).stop 2024-05-03T17:19:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.768+0000 7fe64c96e700 1 -- 172.21.15.67:0/2039754210 shutdown_connections 2024-05-03T17:19:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.768+0000 7fe64c96e700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe6300b2880 0x7fe6300b4d40 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.768+0000 7fe64c96e700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe648108850 0x7fe6481a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.768+0000 7fe64c96e700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe648103750 0x7fe64819bdb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.768+0000 7fe64c96e700 1 --2- 172.21.15.67:0/2039754210 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe648104100 0x7fe64819c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:19:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.768+0000 7fe64c96e700 1 -- 172.21.15.67:0/2039754210 >> 172.21.15.67:0/2039754210 conn(0x7fe64807a600 msgr2=0x7fe648101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:19:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.768+0000 7fe64c96e700 1 -- 172.21.15.67:0/2039754210 shutdown_connections 2024-05-03T17:19:57.770 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:19:57.768+0000 7fe64c96e700 1 -- 172.21.15.67:0/2039754210 wait complete. 2024-05-03T17:19:57.771 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 4 2024-05-03T17:19:58.141 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:57 smithi067 ceph-mon[29731]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:58.141 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:57 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2039754210' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:19:58.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:57 smithi082 ceph-mon[33535]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:58.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:57 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2039754210' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:19:58.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:57 smithi070 ceph-mon[33250]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:58.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:57 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2039754210' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:19:58.453 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 17:19:58 smithi067 systemd[1]: Started Ceph mds.d for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:19:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:59.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:59.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:59 smithi070 ceph-mon[33250]: Deploying daemon mds.e on smithi070 2024-05-03T17:19:59.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:59 smithi070 ceph-mon[33250]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:59.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:59 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.67:6836/583931110,v1:172.21.15.67:6837/583931110] up:boot 2024-05-03T17:19:59.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:59 smithi070 ceph-mon[33250]: fsmap 4 up:standby 2024-05-03T17:19:59.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:19:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-03T17:19:59.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:59.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:59.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:59.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:59.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:59.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:59.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:59 smithi067 ceph-mon[29731]: Deploying daemon mds.e on smithi070 2024-05-03T17:19:59.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:59 smithi067 ceph-mon[29731]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:59.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:59 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.67:6836/583931110,v1:172.21.15.67:6837/583931110] up:boot 2024-05-03T17:19:59.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:59 smithi067 ceph-mon[29731]: fsmap 4 up:standby 2024-05-03T17:19:59.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:19:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-03T17:19:59.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:59.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:59.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:19:59.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:19:59.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:19:59.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:19:59.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:59 smithi082 ceph-mon[33535]: Deploying daemon mds.e on smithi070 2024-05-03T17:19:59.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:59 smithi082 ceph-mon[33535]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:19:59.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:59 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.67:6836/583931110,v1:172.21.15.67:6837/583931110] up:boot 2024-05-03T17:19:59.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:59 smithi082 ceph-mon[33535]: fsmap 4 up:standby 2024-05-03T17:19:59.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:19:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-03T17:20:00.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:00 smithi082 ceph-mon[33535]: overall HEALTH_OK 2024-05-03T17:20:00.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:00 smithi070 ceph-mon[33250]: overall HEALTH_OK 2024-05-03T17:20:00.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:00 smithi067 ceph-mon[29731]: overall HEALTH_OK 2024-05-03T17:20:00.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:20:01.046 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:01.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:01 smithi067 ceph-mon[29731]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:01.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:01 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:01.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:01 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:01.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:01 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:01.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:01 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:01.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:01 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:01.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:01 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:01.539 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:01 smithi070 ceph-mon[33250]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:01.539 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:01 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:01.539 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:01 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:01.539 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:01 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:01.539 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:01 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:01.540 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:01 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:01.540 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:01 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:01.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:01 smithi082 ceph-mon[33535]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:01 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:01 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:01 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:01 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:01 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:01.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:01 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:02.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:02 smithi082 ceph-mon[33535]: Deploying daemon mds.f on smithi082 2024-05-03T17:20:02.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:02 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.70:6834/1966549782,v1:172.21.15.70:6835/1966549782] up:boot 2024-05-03T17:20:02.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:02 smithi082 ceph-mon[33535]: fsmap 5 up:standby 2024-05-03T17:20:02.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:02 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-03T17:20:02.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:02 smithi070 ceph-mon[33250]: Deploying daemon mds.f on smithi082 2024-05-03T17:20:02.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:02 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.70:6834/1966549782,v1:172.21.15.70:6835/1966549782] up:boot 2024-05-03T17:20:02.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:02 smithi070 ceph-mon[33250]: fsmap 5 up:standby 2024-05-03T17:20:02.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:02 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-03T17:20:02.598 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:02 smithi067 ceph-mon[29731]: Deploying daemon mds.f on smithi082 2024-05-03T17:20:02.599 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:02 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.70:6834/1966549782,v1:172.21.15.70:6835/1966549782] up:boot 2024-05-03T17:20:02.599 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:02 smithi067 ceph-mon[29731]: fsmap 5 up:standby 2024-05-03T17:20:02.599 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:02 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-03T17:20:02.973 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.968+0000 7f150c588700 1 -- 172.21.15.67:0/857189583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504075ff0 msgr2=0x7f150410f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.968+0000 7f150c588700 1 --2- 172.21.15.67:0/857189583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504075ff0 0x7f150410f990 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f14f4009990 tx=0x7f14f4009ca0 comp rx=0 tx=0).stop 2024-05-03T17:20:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.969+0000 7f150c588700 1 -- 172.21.15.67:0/857189583 shutdown_connections 2024-05-03T17:20:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.969+0000 7f150c588700 1 --2- 172.21.15.67:0/857189583 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150410fed0 0x7f15041122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.969+0000 7f150c588700 1 --2- 172.21.15.67:0/857189583 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15040756d0 0x7f1504075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.969+0000 7f150c588700 1 --2- 172.21.15.67:0/857189583 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504075ff0 0x7f150410f990 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.969+0000 7f150c588700 1 -- 172.21.15.67:0/857189583 >> 172.21.15.67:0/857189583 conn(0x7f150407a610 msgr2=0x7f150407aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.969+0000 7f150c588700 1 -- 172.21.15.67:0/857189583 shutdown_connections 2024-05-03T17:20:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.969+0000 7f150c588700 1 -- 172.21.15.67:0/857189583 wait complete. 2024-05-03T17:20:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.969+0000 7f150c588700 1 Processor -- start 2024-05-03T17:20:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.969+0000 7f150c588700 1 -- start start 2024-05-03T17:20:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f150c588700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15040756d0 0x7f150419bd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:02.974 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f150c588700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504075ff0 0x7f150419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f150c588700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150410fed0 0x7f15041a0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f150c588700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f15041158b0 con 0x7f1504075ff0 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f150c588700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1504115730 con 0x7f150410fed0 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f150c588700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1504115a30 con 0x7f15040756d0 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f1509b23700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504075ff0 0x7f150419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f1509b23700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504075ff0 0x7f150419c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55292/0 (socket says 172.21.15.67:55292) 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f1509b23700 1 -- 172.21.15.67:0/3674046773 learned_addr learned my addr 172.21.15.67:0/3674046773 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f150ab25700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150410fed0 0x7f15041a0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f150a324700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15040756d0 0x7f150419bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f1509b23700 1 -- 172.21.15.67:0/3674046773 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15040756d0 msgr2=0x7f150419bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f1509b23700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15040756d0 0x7f150419bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f1509b23700 1 -- 172.21.15.67:0/3674046773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150410fed0 msgr2=0x7f15041a0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f1509b23700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150410fed0 0x7f15041a0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f1509b23700 1 -- 172.21.15.67:0/3674046773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15041a0d70 con 0x7f1504075ff0 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f1509b23700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504075ff0 0x7f150419c2a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f14f4000c00 tx=0x7f14f40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:02.975 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f14fb7fe700 1 -- 172.21.15.67:0/3674046773 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f4027070 con 0x7f1504075ff0 2024-05-03T17:20:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f14fb7fe700 1 -- 172.21.15.67:0/3674046773 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14f40040e0 con 0x7f1504075ff0 2024-05-03T17:20:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.970+0000 7f150c588700 1 -- 172.21.15.67:0/3674046773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15041a1000 con 0x7f1504075ff0 2024-05-03T17:20:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.971+0000 7f14fb7fe700 1 -- 172.21.15.67:0/3674046773 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f400f460 con 0x7f1504075ff0 2024-05-03T17:20:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.971+0000 7f150c588700 1 -- 172.21.15.67:0/3674046773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15041a88e0 con 0x7f1504075ff0 2024-05-03T17:20:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.971+0000 7f150c588700 1 -- 172.21.15.67:0/3674046773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f150406a190 con 0x7f1504075ff0 2024-05-03T17:20:02.976 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.973+0000 7f14fb7fe700 1 -- 172.21.15.67:0/3674046773 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f14f402a070 con 0x7f1504075ff0 2024-05-03T17:20:02.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.974+0000 7f14fb7fe700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f14f00b2880 0x7f14f00b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:02.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.974+0000 7f150a324700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f14f00b2880 0x7f14f00b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:02.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.974+0000 7f14fb7fe700 1 -- 172.21.15.67:0/3674046773 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f14f40e31f0 con 0x7f1504075ff0 2024-05-03T17:20:02.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.975+0000 7f150a324700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f14f00b2880 0x7f14f00b4d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1500005fd0 tx=0x7f150000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:02.978 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:02.978+0000 7f14fb7fe700 1 -- 172.21.15.67:0/3674046773 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f14f4031070 con 0x7f1504075ff0 2024-05-03T17:20:03.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.248+0000 7f150c588700 1 -- 172.21.15.67:0/3674046773 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15041a8ab0 con 0x7f1504075ff0 2024-05-03T17:20:03.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.249+0000 7f14fb7fe700 1 -- 172.21.15.67:0/3674046773 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4690 (secure 0 0 0) 0x7f14f4013070 con 0x7f1504075ff0 2024-05-03T17:20:03.250 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:20:03.251 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-03T17:20:03.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.251+0000 7f150c588700 1 -- 172.21.15.67:0/3674046773 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f14f00b2880 msgr2=0x7f14f00b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.251+0000 7f150c588700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f14f00b2880 0x7f14f00b4d40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1500005fd0 tx=0x7f150000a560 comp rx=0 tx=0).stop 2024-05-03T17:20:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.251+0000 7f150c588700 1 -- 172.21.15.67:0/3674046773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504075ff0 msgr2=0x7f150419c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.251+0000 7f150c588700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504075ff0 0x7f150419c2a0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f14f4000c00 tx=0x7f14f40047b0 comp rx=0 tx=0).stop 2024-05-03T17:20:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.251+0000 7f150c588700 1 -- 172.21.15.67:0/3674046773 shutdown_connections 2024-05-03T17:20:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.251+0000 7f150c588700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f14f00b2880 0x7f14f00b4d40 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.251+0000 7f150c588700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f150410fed0 0x7f15041a0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.251+0000 7f150c588700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f15040756d0 0x7f150419bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.251+0000 7f150c588700 1 --2- 172.21.15.67:0/3674046773 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1504075ff0 0x7f150419c2a0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.251+0000 7f150c588700 1 -- 172.21.15.67:0/3674046773 >> 172.21.15.67:0/3674046773 conn(0x7f150407a610 msgr2=0x7f15041006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.251+0000 7f150c588700 1 -- 172.21.15.67:0/3674046773 shutdown_connections 2024-05-03T17:20:03.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:03.251+0000 7f150c588700 1 -- 172.21.15.67:0/3674046773 wait complete. 2024-05-03T17:20:03.253 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 6 2024-05-03T17:20:03.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:03 smithi082 ceph-mon[33535]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:03.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:03 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3674046773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:03.515 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:20:03 smithi082 systemd[1]: Starting Ceph mds.f for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:20:03.669 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:03 smithi067 ceph-mon[29731]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:03.669 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:03 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3674046773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:03.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:03 smithi070 ceph-mon[33250]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:03.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:03 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3674046773' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:03.919 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:20:03 smithi082 podman[64374]: 2024-05-03 17:20:03.531232602 +0000 UTC m=+0.017909582 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:20:03.919 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:20:03 smithi082 podman[64374]: 2024-05-03 17:20:03.650179531 +0000 UTC m=+0.136856511 container create 18fdab0f4e0f5a397c792dc3db7b743d835a4bb8fac54aa66186b6b80dc83f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-f, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T17:20:03.919 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:20:03 smithi082 podman[64374]: 2024-05-03 17:20:03.91936972 +0000 UTC m=+0.406046711 container init 18fdab0f4e0f5a397c792dc3db7b743d835a4bb8fac54aa66186b6b80dc83f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-f, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:20:04.264 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:20:03 smithi082 podman[64374]: 2024-05-03 17:20:03.922118163 +0000 UTC m=+0.408795144 container start 18fdab0f4e0f5a397c792dc3db7b743d835a4bb8fac54aa66186b6b80dc83f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:20:04.265 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:20:03 smithi082 ceph-mds[64393]: -- 172.21.15.82:0/1744428580 <== mon.1 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5634dd6b9a40 con 0x5634dd68cc00 2024-05-03T17:20:04.265 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:20:03 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-f[64389]: starting mds.f at 2024-05-03T17:20:04.265 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:20:04 smithi082 bash[64374]: 18fdab0f4e0f5a397c792dc3db7b743d835a4bb8fac54aa66186b6b80dc83f41 2024-05-03T17:20:04.265 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:20:04 smithi082 systemd[1]: Started Ceph mds.f for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:20:05.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:05 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.82:6834/1568454842,v1:172.21.15.82:6835/1568454842] up:boot 2024-05-03T17:20:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:05 smithi082 ceph-mon[33535]: fsmap 6 up:standby 2024-05-03T17:20:05.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:05 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-03T17:20:05.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:05.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:05.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:05 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.82:6834/1568454842,v1:172.21.15.82:6835/1568454842] up:boot 2024-05-03T17:20:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:05 smithi067 ceph-mon[29731]: fsmap 6 up:standby 2024-05-03T17:20:05.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:05 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-03T17:20:05.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:05.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:05.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:05.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:05.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:05.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:05.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:05 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.82:6834/1568454842,v1:172.21.15.82:6835/1568454842] up:boot 2024-05-03T17:20:05.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:05 smithi070 ceph-mon[33250]: fsmap 6 up:standby 2024-05-03T17:20:05.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:05 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-03T17:20:05.982 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:20:06.246 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:06.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:06 smithi082 ceph-mon[33535]: Deploying daemon mds.c on smithi082 2024-05-03T17:20:06.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:06 smithi082 ceph-mon[33535]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:06.369 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:06 smithi067 ceph-mon[29731]: Deploying daemon mds.c on smithi082 2024-05-03T17:20:06.369 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:06 smithi067 ceph-mon[29731]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:06.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:06 smithi070 ceph-mon[33250]: Deploying daemon mds.c on smithi082 2024-05-03T17:20:06.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:06 smithi070 ceph-mon[33250]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:07.014 INFO:journalctl@ceph.mds.c.smithi082.stdout:May 03 17:20:06 smithi082 systemd[1]: Started Ceph mds.c for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:20:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.926+0000 7efe0ea40700 1 -- 172.21.15.67:0/2364804171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe08102d60 msgr2=0x7efe0810f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.926+0000 7efe0ea40700 1 --2- 172.21.15.67:0/2364804171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe08102d60 0x7efe0810f250 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efe0000b0a0 tx=0x7efe0000b3b0 comp rx=0 tx=0).stop 2024-05-03T17:20:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.926+0000 7efe0ea40700 1 -- 172.21.15.67:0/2364804171 shutdown_connections 2024-05-03T17:20:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.926+0000 7efe0ea40700 1 --2- 172.21.15.67:0/2364804171 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe08102d60 0x7efe0810f250 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.926+0000 7efe0ea40700 1 --2- 172.21.15.67:0/2364804171 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe08108410 0x7efe081087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.926+0000 7efe0ea40700 1 --2- 172.21.15.67:0/2364804171 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe081023a0 0x7efe08102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.926+0000 7efe0ea40700 1 -- 172.21.15.67:0/2364804171 >> 172.21.15.67:0/2364804171 conn(0x7efe08079350 msgr2=0x7efe08079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.926+0000 7efe0ea40700 1 -- 172.21.15.67:0/2364804171 shutdown_connections 2024-05-03T17:20:07.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.926+0000 7efe0ea40700 1 -- 172.21.15.67:0/2364804171 wait complete. 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.927+0000 7efe0ea40700 1 Processor -- start 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.927+0000 7efe0ea40700 1 -- start start 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.927+0000 7efe0ea40700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe081023a0 0x7efe0819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.927+0000 7efe0ea40700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe08102d60 0x7efe0819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.927+0000 7efe0ea40700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe08108410 0x7efe08199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.927+0000 7efe0ea40700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7efe081125b0 con 0x7efe081023a0 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.927+0000 7efe0ea40700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7efe08112430 con 0x7efe08108410 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.927+0000 7efe0ea40700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7efe08112730 con 0x7efe08102d60 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.927+0000 7efe07fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe081023a0 0x7efe0819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe07fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe081023a0 0x7efe0819f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:51068/0 (socket says 172.21.15.67:51068) 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe07fff700 1 -- 172.21.15.67:0/2100537386 learned_addr learned my addr 172.21.15.67:0/2100537386 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe0cfdd700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe08108410 0x7efe08199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe077fe700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe08102d60 0x7efe0819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:07.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe07fff700 1 -- 172.21.15.67:0/2100537386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe08102d60 msgr2=0x7efe0819fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe07fff700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe08102d60 0x7efe0819fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe07fff700 1 -- 172.21.15.67:0/2100537386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe08108410 msgr2=0x7efe08199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe07fff700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe08108410 0x7efe08199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe07fff700 1 -- 172.21.15.67:0/2100537386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0819a0d0 con 0x7efe081023a0 2024-05-03T17:20:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe07fff700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe081023a0 0x7efe0819f7b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efdf800c920 tx=0x7efdf800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe057fa700 1 -- 172.21.15.67:0/2100537386 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8007b00 con 0x7efe081023a0 2024-05-03T17:20:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe0ea40700 1 -- 172.21.15.67:0/2100537386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe0819a3c0 con 0x7efe081023a0 2024-05-03T17:20:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe057fa700 1 -- 172.21.15.67:0/2100537386 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efdf8004ce0 con 0x7efe081023a0 2024-05-03T17:20:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe057fa700 1 -- 172.21.15.67:0/2100537386 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8005500 con 0x7efe081023a0 2024-05-03T17:20:07.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.928+0000 7efe0ea40700 1 -- 172.21.15.67:0/2100537386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe081a6680 con 0x7efe081023a0 2024-05-03T17:20:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.929+0000 7efe0ea40700 1 -- 172.21.15.67:0/2100537386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe0806a190 con 0x7efe081023a0 2024-05-03T17:20:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.931+0000 7efe057fa700 1 -- 172.21.15.67:0/2100537386 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7efdf8007ca0 con 0x7efe081023a0 2024-05-03T17:20:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.931+0000 7efe057fa700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7efdf00b2880 0x7efdf00b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.931+0000 7efe077fe700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7efdf00b2880 0x7efdf00b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:07.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.932+0000 7efe057fa700 1 -- 172.21.15.67:0/2100537386 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7efdf80e1d00 con 0x7efe081023a0 2024-05-03T17:20:07.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.932+0000 7efe077fe700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7efdf00b2880 0x7efdf00b4d40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efdfc006fd0 tx=0x7efdfc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:07.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:07.935+0000 7efe057fa700 1 -- 172.21.15.67:0/2100537386 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7efdf8014030 con 0x7efe081023a0 2024-05-03T17:20:08.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:07 smithi067 ceph-mon[29731]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:08.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:08.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:08.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:08.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:08.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:08.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:08.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:07 smithi067 ceph-mon[29731]: Deploying daemon mds.h on smithi070 2024-05-03T17:20:08.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:07 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.82:6836/370282857,v1:172.21.15.82:6837/370282857] up:boot 2024-05-03T17:20:08.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:07 smithi067 ceph-mon[29731]: fsmap 7 up:standby 2024-05-03T17:20:08.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-03T17:20:08.205 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.205+0000 7efe0ea40700 1 -- 172.21.15.67:0/2100537386 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efe0819aea0 con 0x7efe081023a0 2024-05-03T17:20:08.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.206+0000 7efe057fa700 1 -- 172.21.15.67:0/2100537386 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6337 (secure 0 0 0) 0x7efdf80a9ec0 con 0x7efe081023a0 2024-05-03T17:20:08.206 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:20:08.206 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1568454842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1568454842},{"type":"v1","addr":"172.21.15.82:6835","nonce":1568454842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/370282857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":370282857},{"type":"v1","addr":"172.21.15.82:6837","nonce":370282857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-05-03T17:20:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.208+0000 7efe0ea40700 1 -- 172.21.15.67:0/2100537386 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7efdf00b2880 msgr2=0x7efdf00b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.208+0000 7efe0ea40700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7efdf00b2880 0x7efdf00b4d40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efdfc006fd0 tx=0x7efdfc008040 comp rx=0 tx=0).stop 2024-05-03T17:20:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.208+0000 7efe0ea40700 1 -- 172.21.15.67:0/2100537386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe081023a0 msgr2=0x7efe0819f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.208+0000 7efe0ea40700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe081023a0 0x7efe0819f7b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efdf800c920 tx=0x7efdf800cc30 comp rx=0 tx=0).stop 2024-05-03T17:20:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.208+0000 7efe0ea40700 1 -- 172.21.15.67:0/2100537386 shutdown_connections 2024-05-03T17:20:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.208+0000 7efe0ea40700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7efdf00b2880 0x7efdf00b4d40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.208+0000 7efe0ea40700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7efe08108410 0x7efe08199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.208+0000 7efe0ea40700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe081023a0 0x7efe0819f7b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.208+0000 7efe0ea40700 1 --2- 172.21.15.67:0/2100537386 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7efe08102d60 0x7efe0819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.208+0000 7efe0ea40700 1 -- 172.21.15.67:0/2100537386 >> 172.21.15.67:0/2100537386 conn(0x7efe08079350 msgr2=0x7efe08101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.208+0000 7efe0ea40700 1 -- 172.21.15.67:0/2100537386 shutdown_connections 2024-05-03T17:20:08.209 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:08.208+0000 7efe0ea40700 1 -- 172.21.15.67:0/2100537386 wait complete. 2024-05-03T17:20:08.210 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 8 2024-05-03T17:20:08.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:07 smithi082 ceph-mon[33535]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:08.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:07 smithi082 ceph-mon[33535]: Deploying daemon mds.h on smithi070 2024-05-03T17:20:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:07 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.82:6836/370282857,v1:172.21.15.82:6837/370282857] up:boot 2024-05-03T17:20:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:07 smithi082 ceph-mon[33535]: fsmap 7 up:standby 2024-05-03T17:20:08.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-03T17:20:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:07 smithi070 ceph-mon[33250]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:08.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:08.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:08.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:07 smithi070 ceph-mon[33250]: Deploying daemon mds.h on smithi070 2024-05-03T17:20:08.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:07 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.82:6836/370282857,v1:172.21.15.82:6837/370282857] up:boot 2024-05-03T17:20:08.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:07 smithi070 ceph-mon[33250]: fsmap 7 up:standby 2024-05-03T17:20:08.309 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-03T17:20:09.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:08 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2100537386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:09.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:08 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2100537386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:09.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:08 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2100537386' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:09.308 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:20:09 smithi070 systemd[1]: Starting Ceph mds.h for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:20:09.663 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:20:09 smithi070 podman[63519]: 2024-05-03 17:20:09.296945097 +0000 UTC m=+0.018008782 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:20:09.663 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:20:09 smithi070 podman[63519]: 2024-05-03 17:20:09.410292121 +0000 UTC m=+0.131355802 container create 6563c074bc62b64016e0287c086caf3a15f99926be29869918cac9b91a2bd91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-h, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T17:20:09.663 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:20:09 smithi070 podman[63519]: 2024-05-03 17:20:09.663109071 +0000 UTC m=+0.384172744 container init 6563c074bc62b64016e0287c086caf3a15f99926be29869918cac9b91a2bd91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-h, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:20:09.997 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:20:09 smithi070 podman[63519]: 2024-05-03 17:20:09.66559596 +0000 UTC m=+0.386659642 container start 6563c074bc62b64016e0287c086caf3a15f99926be29869918cac9b91a2bd91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-h, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-05-03T17:20:09.997 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:20:09 smithi070 ceph-mds[63539]: -- 172.21.15.70:0/1082913152 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5563fbea3a40 con 0x5563fbe76c00 2024-05-03T17:20:09.998 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:20:09 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-h[63535]: starting mds.h at 2024-05-03T17:20:09.998 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:20:09 smithi070 bash[63519]: 6563c074bc62b64016e0287c086caf3a15f99926be29869918cac9b91a2bd91e 2024-05-03T17:20:09.998 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:20:09 smithi070 systemd[1]: Started Ceph mds.h for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:20:10.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:09 smithi082 ceph-mon[33535]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:10.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:10.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:10.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:10.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:10.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:10.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:09 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:10.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:09 smithi070 ceph-mon[33250]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:10.316 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:10.316 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:10.316 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:10.316 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:10.317 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:10.317 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:09 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:10.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:09 smithi067 ceph-mon[29731]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:10.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:10.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:10.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:10.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:10.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:10.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:09 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:10.885 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:20:11.147 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:11.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:11 smithi067 ceph-mon[29731]: Deploying daemon mds.i on smithi082 2024-05-03T17:20:11.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:11 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] up:boot 2024-05-03T17:20:11.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:11 smithi067 ceph-mon[29731]: fsmap 8 up:standby 2024-05-03T17:20:11.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:11 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-03T17:20:11.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:11 smithi082 ceph-mon[33535]: Deploying daemon mds.i on smithi082 2024-05-03T17:20:11.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:11 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] up:boot 2024-05-03T17:20:11.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:11 smithi082 ceph-mon[33535]: fsmap 8 up:standby 2024-05-03T17:20:11.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:11 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-03T17:20:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:11 smithi070 ceph-mon[33250]: Deploying daemon mds.i on smithi082 2024-05-03T17:20:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:11 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] up:boot 2024-05-03T17:20:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:11 smithi070 ceph-mon[33250]: fsmap 8 up:standby 2024-05-03T17:20:11.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:11 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-03T17:20:12.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:12 smithi082 ceph-mon[33535]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:12.265 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:20:12 smithi082 systemd[1]: Starting Ceph mds.i for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:20:12.270 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:12 smithi067 ceph-mon[29731]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:12 smithi070 ceph-mon[33250]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:12.571 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:20:12 smithi082 podman[64997]: 2024-05-03 17:20:12.222821546 +0000 UTC m=+0.018269247 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:20:12.571 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:20:12 smithi082 podman[64997]: 2024-05-03 17:20:12.343561763 +0000 UTC m=+0.139009454 container create e05286d474b358251decfe9c59b23e98f794e87906a2a9a940b4a424cd4ea3fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-i, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:20:12.571 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:20:12 smithi082 podman[64997]: 2024-05-03 17:20:12.571172075 +0000 UTC m=+0.366619767 container init e05286d474b358251decfe9c59b23e98f794e87906a2a9a940b4a424cd4ea3fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-i, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-05-03T17:20:12.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.839+0000 7f11f5c47700 1 -- 172.21.15.67:0/3359272419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f0109510 msgr2=0x7f11f01098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:12.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.839+0000 7f11f5c47700 1 --2- 172.21.15.67:0/3359272419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f0109510 0x7f11f01098f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f11e0009990 tx=0x7f11e0009ca0 comp rx=0 tx=0).stop 2024-05-03T17:20:12.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.839+0000 7f11f5c47700 1 -- 172.21.15.67:0/3359272419 shutdown_connections 2024-05-03T17:20:12.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.839+0000 7f11f5c47700 1 --2- 172.21.15.67:0/3359272419 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11f006bb40 0x7f11f0110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:12.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.839+0000 7f11f5c47700 1 --2- 172.21.15.67:0/3359272419 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f0109510 0x7f11f01098f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:12.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.839+0000 7f11f5c47700 1 --2- 172.21.15.67:0/3359272419 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11f006b180 0x7f11f006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:12.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.839+0000 7f11f5c47700 1 -- 172.21.15.67:0/3359272419 >> 172.21.15.67:0/3359272419 conn(0x7f11f0079350 msgr2=0x7f11f0079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:12.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.839+0000 7f11f5c47700 1 -- 172.21.15.67:0/3359272419 shutdown_connections 2024-05-03T17:20:12.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.839+0000 7f11f5c47700 1 -- 172.21.15.67:0/3359272419 wait complete. 2024-05-03T17:20:12.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.841+0000 7f11f5c47700 1 Processor -- start 2024-05-03T17:20:12.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.841+0000 7f11f5c47700 1 -- start start 2024-05-03T17:20:12.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.841+0000 7f11f5c47700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f006b180 0x7f11f0102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:12.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.841+0000 7f11f5c47700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11f006bb40 0x7f11f01029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:12.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.841+0000 7f11f5c47700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11f0109510 0x7f11f0102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:12.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.841+0000 7f11f5c47700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f11f0113640 con 0x7f11f006b180 2024-05-03T17:20:12.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.841+0000 7f11f5c47700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f11f01134c0 con 0x7f11f0109510 2024-05-03T17:20:12.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.841+0000 7f11f5c47700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f11f01137c0 con 0x7f11f006bb40 2024-05-03T17:20:12.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.841+0000 7f11ef7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f006b180 0x7f11f0102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.841+0000 7f11eeffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11f006bb40 0x7f11f01029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.841+0000 7f11eeffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11f006bb40 0x7f11f01029b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:47100/0 (socket says 172.21.15.67:47100) 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.841+0000 7f11eeffd700 1 -- 172.21.15.67:0/2122085400 learned_addr learned my addr 172.21.15.67:0/2122085400 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.842+0000 7f11effff700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11f0109510 0x7f11f0102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.842+0000 7f11eeffd700 1 -- 172.21.15.67:0/2122085400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11f0109510 msgr2=0x7f11f0102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.842+0000 7f11eeffd700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11f0109510 0x7f11f0102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.842+0000 7f11eeffd700 1 -- 172.21.15.67:0/2122085400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f006b180 msgr2=0x7f11f0102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.842+0000 7f11eeffd700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f006b180 0x7f11f0102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.842+0000 7f11eeffd700 1 -- 172.21.15.67:0/2122085400 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f01a8090 con 0x7f11f006bb40 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.842+0000 7f11eeffd700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11f006bb40 0x7f11f01029b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f11e400e9c0 tx=0x7f11e400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.842+0000 7f11ecff9700 1 -- 172.21.15.67:0/2122085400 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e400cbd0 con 0x7f11f006bb40 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.842+0000 7f11f5c47700 1 -- 172.21.15.67:0/2122085400 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f01a8380 con 0x7f11f006bb40 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.842+0000 7f11f5c47700 1 -- 172.21.15.67:0/2122085400 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f01a88c0 con 0x7f11f006bb40 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.842+0000 7f11ecff9700 1 -- 172.21.15.67:0/2122085400 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f11e400cd70 con 0x7f11f006bb40 2024-05-03T17:20:12.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.842+0000 7f11ecff9700 1 -- 172.21.15.67:0/2122085400 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e4005610 con 0x7f11f006bb40 2024-05-03T17:20:12.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.843+0000 7f11f5c47700 1 -- 172.21.15.67:0/2122085400 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11dc005330 con 0x7f11f006bb40 2024-05-03T17:20:12.845 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.844+0000 7f11ecff9700 1 -- 172.21.15.67:0/2122085400 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f11e4004830 con 0x7f11f006bb40 2024-05-03T17:20:12.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.845+0000 7f11ecff9700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f11d80b2900 0x7f11d80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:12.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.846+0000 7f11ef7fe700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f11d80b2900 0x7f11d80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:12.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.846+0000 7f11ecff9700 1 -- 172.21.15.67:0/2122085400 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f11e40e1f70 con 0x7f11f006bb40 2024-05-03T17:20:12.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.846+0000 7f11ef7fe700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f11d80b2900 0x7f11d80b4dc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f11e00098e0 tx=0x7f11e0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:12.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:12.850+0000 7f11ecff9700 1 -- 172.21.15.67:0/2122085400 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f11e40aa090 con 0x7f11f006bb40 2024-05-03T17:20:13.014 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:20:12 smithi082 podman[64997]: 2024-05-03 17:20:12.573949632 +0000 UTC m=+0.369397325 container start e05286d474b358251decfe9c59b23e98f794e87906a2a9a940b4a424cd4ea3fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-i, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430) 2024-05-03T17:20:13.015 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:20:12 smithi082 ceph-mds[65016]: -- 172.21.15.82:0/1043539167 <== mon.1 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563aabc81a40 con 0x563aabc54800 2024-05-03T17:20:13.015 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:20:12 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-i[65012]: starting mds.i at 2024-05-03T17:20:13.015 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:20:12 smithi082 bash[64997]: e05286d474b358251decfe9c59b23e98f794e87906a2a9a940b4a424cd4ea3fc 2024-05-03T17:20:13.015 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:20:12 smithi082 systemd[1]: Started Ceph mds.i for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:20:13.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.118+0000 7f11f5c47700 1 -- 172.21.15.67:0/2122085400 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11dc005620 con 0x7f11f006bb40 2024-05-03T17:20:13.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.119+0000 7f11ecff9700 1 -- 172.21.15.67:0/2122085400 <== mon.1 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7989 (secure 0 0 0) 0x7f11e40a9990 con 0x7f11f006bb40 2024-05-03T17:20:13.120 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:20:13.120 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1568454842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1568454842},{"type":"v1","addr":"172.21.15.82:6835","nonce":1568454842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/370282857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":370282857},{"type":"v1","addr":"172.21.15.82:6837","nonce":370282857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24442,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6839/2084218047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2084218047},{"type":"v1","addr":"172.21.15.82:6839","nonce":2084218047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3323713018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3323713018},{"type":"v1","addr":"172.21.15.70:6837","nonce":3323713018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-03T17:20:13.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.121+0000 7f11f5c47700 1 -- 172.21.15.67:0/2122085400 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f11d80b2900 msgr2=0x7f11d80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:13.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.121+0000 7f11f5c47700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f11d80b2900 0x7f11d80b4dc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f11e00098e0 tx=0x7f11e0005960 comp rx=0 tx=0).stop 2024-05-03T17:20:13.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.121+0000 7f11f5c47700 1 -- 172.21.15.67:0/2122085400 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11f006bb40 msgr2=0x7f11f01029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:13.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.121+0000 7f11f5c47700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11f006bb40 0x7f11f01029b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f11e400e9c0 tx=0x7f11e400ecd0 comp rx=0 tx=0).stop 2024-05-03T17:20:13.122 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.122+0000 7f11f5c47700 1 -- 172.21.15.67:0/2122085400 shutdown_connections 2024-05-03T17:20:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.122+0000 7f11f5c47700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f11d80b2900 0x7f11d80b4dc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.122+0000 7f11f5c47700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f11f0109510 0x7f11f0102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.122+0000 7f11f5c47700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f11f006b180 0x7f11f0102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.122+0000 7f11f5c47700 1 --2- 172.21.15.67:0/2122085400 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f11f006bb40 0x7f11f01029b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.122+0000 7f11f5c47700 1 -- 172.21.15.67:0/2122085400 >> 172.21.15.67:0/2122085400 conn(0x7f11f0079350 msgr2=0x7f11f01010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.122+0000 7f11f5c47700 1 -- 172.21.15.67:0/2122085400 shutdown_connections 2024-05-03T17:20:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:13.122+0000 7f11f5c47700 1 -- 172.21.15.67:0/2122085400 wait complete. 2024-05-03T17:20:13.123 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 10 2024-05-03T17:20:13.813 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:13 smithi067 ceph-mon[29731]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:13.813 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:13.813 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:13.813 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:13.813 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:13.813 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:13.813 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:13.813 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:13 smithi067 ceph-mon[29731]: Deploying daemon mds.g on smithi067 2024-05-03T17:20:13.813 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:13 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] up:boot 2024-05-03T17:20:13.814 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:13 smithi067 ceph-mon[29731]: fsmap 9 up:standby 2024-05-03T17:20:13.814 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:13 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-03T17:20:13.814 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:13 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2122085400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:14.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:13 smithi082 ceph-mon[33535]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:14.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:14.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:14.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:14.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:14.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:14.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:14.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:13 smithi082 ceph-mon[33535]: Deploying daemon mds.g on smithi067 2024-05-03T17:20:14.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:13 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] up:boot 2024-05-03T17:20:14.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:13 smithi082 ceph-mon[33535]: fsmap 9 up:standby 2024-05-03T17:20:14.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:13 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-03T17:20:14.016 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:13 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2122085400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:13 smithi070 ceph-mon[33250]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:13 smithi070 ceph-mon[33250]: Deploying daemon mds.g on smithi067 2024-05-03T17:20:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:13 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] up:boot 2024-05-03T17:20:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:13 smithi070 ceph-mon[33250]: fsmap 9 up:standby 2024-05-03T17:20:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:13 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-03T17:20:14.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:13 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2122085400' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:15.703 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:20:15 smithi067 systemd[1]: Starting Ceph mds.g for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:20:15.849 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:20:15.969 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:15 smithi067 ceph-mon[29731]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:15.969 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:20:15 smithi067 podman[71974]: 2024-05-03 17:20:15.607529054 +0000 UTC m=+0.018546825 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:20:15.969 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:20:15 smithi067 podman[71974]: 2024-05-03 17:20:15.706197296 +0000 UTC m=+0.117215065 container create 08c0c8ab3a741232de03883df2e0c41f8c51ded697cdff119cc202506630bdfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-g, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-03T17:20:15.970 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:20:15 smithi067 podman[71974]: 2024-05-03 17:20:15.942427412 +0000 UTC m=+0.353445182 container init 08c0c8ab3a741232de03883df2e0c41f8c51ded697cdff119cc202506630bdfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-g, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:20:15.970 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:20:15 smithi067 podman[71974]: 2024-05-03 17:20:15.945326392 +0000 UTC m=+0.356344161 container start 08c0c8ab3a741232de03883df2e0c41f8c51ded697cdff119cc202506630bdfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-05-03T17:20:15.970 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:20:15 smithi067 ceph-mds[72011]: -- 172.21.15.67:0/2224221199 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5599edda1a40 con 0x5599edd74c00 2024-05-03T17:20:16.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:15 smithi082 ceph-mon[33535]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:16.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:15 smithi070 ceph-mon[33250]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:16.117 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:16.241 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:20:15 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-g[71989]: starting mds.g at 2024-05-03T17:20:16.241 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:20:16 smithi067 bash[71974]: 08c0c8ab3a741232de03883df2e0c41f8c51ded697cdff119cc202506630bdfc 2024-05-03T17:20:16.241 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:20:16 smithi067 systemd[1]: Started Ceph mds.g for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:20:17.351 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:17.352 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:17.352 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:17.352 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:17.352 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:17.352 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:17.352 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:17 smithi067 ceph-mon[29731]: Deploying daemon mds.a on smithi067 2024-05-03T17:20:17.352 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:17 smithi067 ceph-mon[29731]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:17.352 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:17 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.67:6838/316463706,v1:172.21.15.67:6839/316463706] up:boot 2024-05-03T17:20:17.352 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:17 smithi067 ceph-mon[29731]: fsmap 10 up:standby 2024-05-03T17:20:17.352 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:17 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-03T17:20:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:17 smithi082 ceph-mon[33535]: Deploying daemon mds.a on smithi067 2024-05-03T17:20:17.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:17 smithi082 ceph-mon[33535]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:17.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:17 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.67:6838/316463706,v1:172.21.15.67:6839/316463706] up:boot 2024-05-03T17:20:17.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:17 smithi082 ceph-mon[33535]: fsmap 10 up:standby 2024-05-03T17:20:17.516 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:17 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-03T17:20:17.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:17.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:17.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:17.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:17.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:17.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:17.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:17 smithi070 ceph-mon[33250]: Deploying daemon mds.a on smithi067 2024-05-03T17:20:17.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:17 smithi070 ceph-mon[33250]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:17.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:17 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.67:6838/316463706,v1:172.21.15.67:6839/316463706] up:boot 2024-05-03T17:20:17.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:17 smithi070 ceph-mon[33250]: fsmap 10 up:standby 2024-05-03T17:20:17.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:17 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-03T17:20:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.882+0000 7f45ad3f4700 1 -- 172.21.15.67:0/2252360138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a8101c50 msgr2=0x7f45a8102030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.882+0000 7f45ad3f4700 1 --2- 172.21.15.67:0/2252360138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a8101c50 0x7f45a8102030 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f45980099c0 tx=0x7f4598009cd0 comp rx=0 tx=0).stop 2024-05-03T17:20:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.882+0000 7f45ad3f4700 1 -- 172.21.15.67:0/2252360138 shutdown_connections 2024-05-03T17:20:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.882+0000 7f45ad3f4700 1 --2- 172.21.15.67:0/2252360138 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a8108890 0x7f45a8110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.882+0000 7f45ad3f4700 1 --2- 172.21.15.67:0/2252360138 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a8101c50 0x7f45a8102030 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.882+0000 7f45ad3f4700 1 --2- 172.21.15.67:0/2252360138 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45a8102600 0x7f45a8108160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.882+0000 7f45ad3f4700 1 -- 172.21.15.67:0/2252360138 >> 172.21.15.67:0/2252360138 conn(0x7f45a80789a0 msgr2=0x7f45a80ffd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:17.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.882+0000 7f45ad3f4700 1 -- 172.21.15.67:0/2252360138 shutdown_connections 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.882+0000 7f45ad3f4700 1 -- 172.21.15.67:0/2252360138 wait complete. 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.883+0000 7f45ad3f4700 1 Processor -- start 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.883+0000 7f45ad3f4700 1 -- start start 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.883+0000 7f45ad3f4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a8101c50 0x7f45a8102bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.883+0000 7f45ad3f4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45a8102600 0x7f45a8103110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.883+0000 7f45ad3f4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a8108890 0x7f45a8103650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.883+0000 7f45ad3f4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f45a8113680 con 0x7f45a8108890 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.883+0000 7f45ad3f4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f45a8113500 con 0x7f45a8102600 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.883+0000 7f45ad3f4700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f45a8113800 con 0x7f45a8101c50 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45a77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a8108890 0x7f45a8103650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45a6ffd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a8101c50 0x7f45a8102bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45a77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a8108890 0x7f45a8103650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57736/0 (socket says 172.21.15.67:57736) 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45a77fe700 1 -- 172.21.15.67:0/465330254 learned_addr learned my addr 172.21.15.67:0/465330254 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45a67fc700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45a8102600 0x7f45a8103110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:17.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45a77fe700 1 -- 172.21.15.67:0/465330254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a8101c50 msgr2=0x7f45a8102bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:17.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45a77fe700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a8101c50 0x7f45a8102bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:17.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45a77fe700 1 -- 172.21.15.67:0/465330254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45a8102600 msgr2=0x7f45a8103110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:17.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45a77fe700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45a8102600 0x7f45a8103110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:17.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45a77fe700 1 -- 172.21.15.67:0/465330254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a81a7ff0 con 0x7f45a8108890 2024-05-03T17:20:17.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45a77fe700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a8108890 0x7f45a8103650 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f45a0007f10 tx=0x7f45a000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:17.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f458ffff700 1 -- 172.21.15.67:0/465330254 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a0009e20 con 0x7f45a8108890 2024-05-03T17:20:17.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45ad3f4700 1 -- 172.21.15.67:0/465330254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a81a82e0 con 0x7f45a8108890 2024-05-03T17:20:17.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f458ffff700 1 -- 172.21.15.67:0/465330254 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45a0004bb0 con 0x7f45a8108890 2024-05-03T17:20:17.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f458ffff700 1 -- 172.21.15.67:0/465330254 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a00058e0 con 0x7f45a8108890 2024-05-03T17:20:17.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.884+0000 7f45ad3f4700 1 -- 172.21.15.67:0/465330254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a81a8820 con 0x7f45a8108890 2024-05-03T17:20:17.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.885+0000 7f45ad3f4700 1 -- 172.21.15.67:0/465330254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45a810bad0 con 0x7f45a8108890 2024-05-03T17:20:17.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.887+0000 7f458ffff700 1 -- 172.21.15.67:0/465330254 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f45a001a020 con 0x7f45a8108890 2024-05-03T17:20:17.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.887+0000 7f458ffff700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45900b2880 0x7f45900b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:17.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.888+0000 7f45a6ffd700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45900b2880 0x7f45900b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:17.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.888+0000 7f458ffff700 1 -- 172.21.15.67:0/465330254 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f45a00e19a0 con 0x7f45a8108890 2024-05-03T17:20:17.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.888+0000 7f45a6ffd700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45900b2880 0x7f45900b4d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4598009990 tx=0x7f459800b790 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:17.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:17.891+0000 7f458ffff700 1 -- 172.21.15.67:0/465330254 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f45a0010070 con 0x7f45a8108890 2024-05-03T17:20:18.167 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.167+0000 7f45ad3f4700 1 -- 172.21.15.67:0/465330254 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45a810b4b0 con 0x7f45a8108890 2024-05-03T17:20:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.167+0000 7f458ffff700 1 -- 172.21.15.67:0/465330254 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8812 (secure 0 0 0) 0x7f45a0004d60 con 0x7f45a8108890 2024-05-03T17:20:18.168 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:20:18.168 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/316463706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":316463706},{"type":"v1","addr":"172.21.15.67:6839","nonce":316463706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1568454842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1568454842},{"type":"v1","addr":"172.21.15.82:6835","nonce":1568454842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/370282857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":370282857},{"type":"v1","addr":"172.21.15.82:6837","nonce":370282857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24442,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6839/2084218047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2084218047},{"type":"v1","addr":"172.21.15.82:6839","nonce":2084218047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3323713018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3323713018},{"type":"v1","addr":"172.21.15.70:6837","nonce":3323713018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-03T17:20:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.169+0000 7f45ad3f4700 1 -- 172.21.15.67:0/465330254 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45900b2880 msgr2=0x7f45900b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.169+0000 7f45ad3f4700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45900b2880 0x7f45900b4d40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4598009990 tx=0x7f459800b790 comp rx=0 tx=0).stop 2024-05-03T17:20:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.169+0000 7f45ad3f4700 1 -- 172.21.15.67:0/465330254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a8108890 msgr2=0x7f45a8103650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.169+0000 7f45ad3f4700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a8108890 0x7f45a8103650 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f45a0007f10 tx=0x7f45a000d3b0 comp rx=0 tx=0).stop 2024-05-03T17:20:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.170+0000 7f45ad3f4700 1 -- 172.21.15.67:0/465330254 shutdown_connections 2024-05-03T17:20:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.170+0000 7f45ad3f4700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45900b2880 0x7f45900b4d40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.170+0000 7f45ad3f4700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f45a8108890 0x7f45a8103650 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.170+0000 7f45ad3f4700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f45a8101c50 0x7f45a8102bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.170+0000 7f45ad3f4700 1 --2- 172.21.15.67:0/465330254 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f45a8102600 0x7f45a8103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.170+0000 7f45ad3f4700 1 -- 172.21.15.67:0/465330254 >> 172.21.15.67:0/465330254 conn(0x7f45a80789a0 msgr2=0x7f45a8107000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.170+0000 7f45ad3f4700 1 -- 172.21.15.67:0/465330254 shutdown_connections 2024-05-03T17:20:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:18.170+0000 7f45ad3f4700 1 -- 172.21.15.67:0/465330254 wait complete. 2024-05-03T17:20:18.171 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 11 2024-05-03T17:20:18.242 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:18 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/465330254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:18.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:18 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/465330254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:18.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:18 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/465330254' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:19.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:19 smithi067 ceph-mon[29731]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:19.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:19 smithi082 ceph-mon[33535]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:19.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:19 smithi070 ceph-mon[33250]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:20.703 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 17:20:20 smithi067 systemd[1]: Started Ceph mds.a for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:20:21.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:21 smithi082 ceph-mon[33535]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:21.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:21 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.67:6840/3422086111,v1:172.21.15.67:6841/3422086111] up:boot 2024-05-03T17:20:21.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:21 smithi082 ceph-mon[33535]: fsmap 11 up:standby 2024-05-03T17:20:21.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-03T17:20:21.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:21.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:21.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:21.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:21.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:21.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:21 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:21.766 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:21 smithi082 ceph-mon[33535]: Deploying daemon mds.b on smithi070 2024-05-03T17:20:21.786 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:21 smithi070 ceph-mon[33250]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:21.786 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:21 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.67:6840/3422086111,v1:172.21.15.67:6841/3422086111] up:boot 2024-05-03T17:20:21.786 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:21 smithi070 ceph-mon[33250]: fsmap 11 up:standby 2024-05-03T17:20:21.786 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-03T17:20:21.786 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:21.786 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:21.786 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:21.786 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:21.786 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:21.786 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:21 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:21.787 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:21 smithi070 ceph-mon[33250]: Deploying daemon mds.b on smithi070 2024-05-03T17:20:21.874 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:20:21.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:21 smithi067 ceph-mon[29731]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:21.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:21 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.67:6840/3422086111,v1:172.21.15.67:6841/3422086111] up:boot 2024-05-03T17:20:21.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:21 smithi067 ceph-mon[29731]: fsmap 11 up:standby 2024-05-03T17:20:21.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-03T17:20:21.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:21.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:21.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:21.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-03T17:20:21.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-03T17:20:21.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:21 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:21.902 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:21 smithi067 ceph-mon[29731]: Deploying daemon mds.b on smithi070 2024-05-03T17:20:22.141 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:22.808 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:20:22 smithi070 systemd[1]: Starting Ceph mds.b for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:20:23.267 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:20:22 smithi070 podman[63840]: 2024-05-03 17:20:22.955605554 +0000 UTC m=+0.110432517 container create d4ff0e3af46a6ac44017927f2c1728869f001a04e7a531814fa862f42470cc80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:20:23.267 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:20:22 smithi070 podman[63840]: 2024-05-03 17:20:22.863340351 +0000 UTC m=+0.018167285 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:20:23.267 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:20:23 smithi070 podman[63840]: 2024-05-03 17:20:23.174760563 +0000 UTC m=+0.329587481 container init d4ff0e3af46a6ac44017927f2c1728869f001a04e7a531814fa862f42470cc80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-03T17:20:23.267 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:20:23 smithi070 podman[63840]: 2024-05-03 17:20:23.177263541 +0000 UTC m=+0.332090462 container start d4ff0e3af46a6ac44017927f2c1728869f001a04e7a531814fa862f42470cc80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:20:23.267 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:20:23 smithi070 ceph-mds[63859]: -- 172.21.15.70:0/3471448175 <== mon.2 v2:172.21.15.70:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b467adba40 con 0x55b467aae800 2024-05-03T17:20:23.267 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:20:23 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b[63855]: starting mds.b at 2024-05-03T17:20:23.267 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:20:23 smithi070 bash[63840]: d4ff0e3af46a6ac44017927f2c1728869f001a04e7a531814fa862f42470cc80 2024-05-03T17:20:23.558 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:20:23 smithi070 systemd[1]: Started Ceph mds.b for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:20:23.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:23 smithi070 ceph-mon[33250]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:23.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:23 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.639 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:23 smithi067 ceph-mon[29731]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:23.639 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.639 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.639 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.639 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.639 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:23 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:23 smithi082 ceph-mon[33535]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:23.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:23 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:23.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.885+0000 7f7a495fa700 1 -- 172.21.15.67:0/3060400677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a44102820 msgr2=0x7f7a44110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.885+0000 7f7a495fa700 1 --2- 172.21.15.67:0/3060400677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a44102820 0x7f7a44110290 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c00b0a0 tx=0x7f7a3c00b3b0 comp rx=0 tx=0).stop 2024-05-03T17:20:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.885+0000 7f7a495fa700 1 -- 172.21.15.67:0/3060400677 shutdown_connections 2024-05-03T17:20:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.885+0000 7f7a495fa700 1 --2- 172.21.15.67:0/3060400677 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a44102820 0x7f7a44110290 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.886+0000 7f7a495fa700 1 --2- 172.21.15.67:0/3060400677 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7a44108580 0x7f7a44108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.886+0000 7f7a495fa700 1 --2- 172.21.15.67:0/3060400677 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a44101e60 0x7f7a441022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.886+0000 7f7a495fa700 1 -- 172.21.15.67:0/3060400677 >> 172.21.15.67:0/3060400677 conn(0x7f7a440776c0 msgr2=0x7f7a44077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.886+0000 7f7a495fa700 1 -- 172.21.15.67:0/3060400677 shutdown_connections 2024-05-03T17:20:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.886+0000 7f7a495fa700 1 -- 172.21.15.67:0/3060400677 wait complete. 2024-05-03T17:20:23.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.886+0000 7f7a495fa700 1 Processor -- start 2024-05-03T17:20:23.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.886+0000 7f7a495fa700 1 -- start start 2024-05-03T17:20:23.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a495fa700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7a44101e60 0x7f7a44103040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a495fa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a44102820 0x7f7a44103580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a495fa700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a44108580 0x7f7a44103ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a495fa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7a44113670 con 0x7f7a44102820 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a495fa700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7a441134f0 con 0x7f7a44108580 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a495fa700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7a441137f0 con 0x7f7a44101e60 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a427fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a44102820 0x7f7a44103580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a427fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a44102820 0x7f7a44103580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57766/0 (socket says 172.21.15.67:57766) 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a427fc700 1 -- 172.21.15.67:0/3938000093 learned_addr learned my addr 172.21.15.67:0/3938000093 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a42ffd700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7a44101e60 0x7f7a44103040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a437fe700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a44108580 0x7f7a44103ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a427fc700 1 -- 172.21.15.67:0/3938000093 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7a44101e60 msgr2=0x7f7a44103040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a427fc700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7a44101e60 0x7f7a44103040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a427fc700 1 -- 172.21.15.67:0/3938000093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a44108580 msgr2=0x7f7a44103ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:23.889 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a427fc700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a44108580 0x7f7a44103ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a427fc700 1 -- 172.21.15.67:0/3938000093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a441a7fc0 con 0x7f7a44102820 2024-05-03T17:20:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a437fe700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a44108580 0x7f7a44103ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:20:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a427fc700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a44102820 0x7f7a44103580 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800e9a0 tx=0x7f7a3800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.887+0000 7f7a2bfff700 1 -- 172.21.15.67:0/3938000093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3800ca10 con 0x7f7a44102820 2024-05-03T17:20:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.888+0000 7f7a495fa700 1 -- 172.21.15.67:0/3938000093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a441a82b0 con 0x7f7a44102820 2024-05-03T17:20:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.888+0000 7f7a2bfff700 1 -- 172.21.15.67:0/3938000093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7a3800cbb0 con 0x7f7a44102820 2024-05-03T17:20:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.888+0000 7f7a2bfff700 1 -- 172.21.15.67:0/3938000093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a38018680 con 0x7f7a44102820 2024-05-03T17:20:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.888+0000 7f7a495fa700 1 -- 172.21.15.67:0/3938000093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a441a8840 con 0x7f7a44102820 2024-05-03T17:20:23.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.889+0000 7f7a495fa700 1 -- 172.21.15.67:0/3938000093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a4406a190 con 0x7f7a44102820 2024-05-03T17:20:23.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.890+0000 7f7a2bfff700 1 -- 172.21.15.67:0/3938000093 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f7a38010430 con 0x7f7a44102820 2024-05-03T17:20:23.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.891+0000 7f7a2bfff700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f7a2c0b2880 0x7f7a2c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:23.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.891+0000 7f7a42ffd700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f7a2c0b2880 0x7f7a2c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:23.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.891+0000 7f7a2bfff700 1 -- 172.21.15.67:0/3938000093 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f7a380e1f60 con 0x7f7a44102820 2024-05-03T17:20:23.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.892+0000 7f7a42ffd700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f7a2c0b2880 0x7f7a2c0b4d40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7a44078c40 tx=0x7f7a34009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:23.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:23.896+0000 7f7a2bfff700 1 -- 172.21.15.67:0/3938000093 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7a380aa080 con 0x7f7a44102820 2024-05-03T17:20:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.182+0000 7f7a495fa700 1 -- 172.21.15.67:0/3938000093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7a4402e6d0 con 0x7f7a44102820 2024-05-03T17:20:24.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.183+0000 7f7a2bfff700 1 -- 172.21.15.67:0/3938000093 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10461 (secure 0 0 0) 0x7f7a380b2100 con 0x7f7a44102820 2024-05-03T17:20:24.184 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:20:24.184 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/316463706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":316463706},{"type":"v1","addr":"172.21.15.67:6839","nonce":316463706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3422086111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3422086111},{"type":"v1","addr":"172.21.15.67:6841","nonce":3422086111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24421,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6835/1568454842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1568454842},{"type":"v1","addr":"172.21.15.82:6835","nonce":1568454842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24425,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6837/370282857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":370282857},{"type":"v1","addr":"172.21.15.82:6837","nonce":370282857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24442,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6839/2084218047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2084218047},{"type":"v1","addr":"172.21.15.82:6839","nonce":2084218047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24446,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6837/3323713018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3323713018},{"type":"v1","addr":"172.21.15.70:6837","nonce":3323713018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24464,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6839/902276547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":902276547},{"type":"v1","addr":"172.21.15.70:6839","nonce":902276547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-03T17:20:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.185+0000 7f7a495fa700 1 -- 172.21.15.67:0/3938000093 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f7a2c0b2880 msgr2=0x7f7a2c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:24.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.185+0000 7f7a495fa700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f7a2c0b2880 0x7f7a2c0b4d40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f7a44078c40 tx=0x7f7a34009260 comp rx=0 tx=0).stop 2024-05-03T17:20:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.185+0000 7f7a495fa700 1 -- 172.21.15.67:0/3938000093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a44102820 msgr2=0x7f7a44103580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.185+0000 7f7a495fa700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a44102820 0x7f7a44103580 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800e9a0 tx=0x7f7a3800ecb0 comp rx=0 tx=0).stop 2024-05-03T17:20:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.185+0000 7f7a495fa700 1 -- 172.21.15.67:0/3938000093 shutdown_connections 2024-05-03T17:20:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.185+0000 7f7a495fa700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f7a2c0b2880 0x7f7a2c0b4d40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.185+0000 7f7a495fa700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7a44108580 0x7f7a44103ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.185+0000 7f7a495fa700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7a44101e60 0x7f7a44103040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.185+0000 7f7a495fa700 1 --2- 172.21.15.67:0/3938000093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7a44102820 0x7f7a44103580 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.185+0000 7f7a495fa700 1 -- 172.21.15.67:0/3938000093 >> 172.21.15.67:0/3938000093 conn(0x7f7a440776c0 msgr2=0x7f7a44100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.185+0000 7f7a495fa700 1 -- 172.21.15.67:0/3938000093 shutdown_connections 2024-05-03T17:20:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:24.185+0000 7f7a495fa700 1 -- 172.21.15.67:0/3938000093 wait complete. 2024-05-03T17:20:24.186 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 13 2024-05-03T17:20:24.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:24 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] up:boot 2024-05-03T17:20:24.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:24 smithi070 ceph-mon[33250]: fsmap 12 up:standby 2024-05-03T17:20:24.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-03T17:20:24.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:24 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:20:24.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:24 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3938000093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:24.581 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:24 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] up:boot 2024-05-03T17:20:24.581 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:24 smithi082 ceph-mon[33535]: fsmap 12 up:standby 2024-05-03T17:20:24.581 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-03T17:20:24.582 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:24 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:20:24.582 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:24 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3938000093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:24.737 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:24 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] up:boot 2024-05-03T17:20:24.737 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:24 smithi067 ceph-mon[29731]: fsmap 12 up:standby 2024-05-03T17:20:24.737 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-03T17:20:24.737 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:24 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:20:24.737 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:24 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3938000093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:20:25.515 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-05-03T17:20:25.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:25 smithi082 ceph-mon[33535]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:25.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:25.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:25.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:25.765 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:25 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:25.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:25 smithi070 ceph-mon[33250]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:25.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:25.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:25.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:25.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:25 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:25.819 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:25.945 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:25 smithi067 ceph-mon[29731]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:25.945 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:25.945 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:25.945 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:25.945 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:25 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:26 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:26 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:26 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:26 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:27.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:26 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:20:27.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:26 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:26 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:20:27.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:26 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:26 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:26 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:26 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:27.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:26 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:20:27.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:26 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:26 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:20:27.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:26 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:26 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:26 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:26 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:27.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:26 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:20:27.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:26 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:27.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:26 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:20:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.613+0000 7f626fe5c700 1 -- 172.21.15.67:0/3369312037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6268103750 msgr2=0x7f6268103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.613+0000 7f626fe5c700 1 --2- 172.21.15.67:0/3369312037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6268103750 0x7f6268103b30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f625c0099c0 tx=0x7f625c009cd0 comp rx=0 tx=0).stop 2024-05-03T17:20:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.613+0000 7f626fe5c700 1 -- 172.21.15.67:0/3369312037 shutdown_connections 2024-05-03T17:20:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.613+0000 7f626fe5c700 1 --2- 172.21.15.67:0/3369312037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6268108850 0x7f62681103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.613+0000 7f626fe5c700 1 --2- 172.21.15.67:0/3369312037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6268103750 0x7f6268103b30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.613+0000 7f626fe5c700 1 --2- 172.21.15.67:0/3369312037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268104100 0x7f6268108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.613+0000 7f626fe5c700 1 -- 172.21.15.67:0/3369312037 >> 172.21.15.67:0/3369312037 conn(0x7f626807a600 msgr2=0x7f626807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.613+0000 7f626fe5c700 1 -- 172.21.15.67:0/3369312037 shutdown_connections 2024-05-03T17:20:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.613+0000 7f626fe5c700 1 -- 172.21.15.67:0/3369312037 wait complete. 2024-05-03T17:20:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626fe5c700 1 Processor -- start 2024-05-03T17:20:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626fe5c700 1 -- start start 2024-05-03T17:20:27.625 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626fe5c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268103750 0x7f626819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626fe5c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6268104100 0x7f626819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626fe5c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6268108850 0x7f6268199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626fe5c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6268113690 con 0x7f6268108850 2024-05-03T17:20:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626fe5c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6268113510 con 0x7f6268103750 2024-05-03T17:20:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626fe5c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6268113810 con 0x7f6268104100 2024-05-03T17:20:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626e3f9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6268108850 0x7f6268199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626dbf8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268103750 0x7f626819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626dbf8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268103750 0x7f626819f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.67:37524/0 (socket says 172.21.15.67:37524) 2024-05-03T17:20:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626dbf8700 1 -- 172.21.15.67:0/769231158 learned_addr learned my addr 172.21.15.67:0/769231158 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.614+0000 7f626d3f7700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6268104100 0x7f626819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.615+0000 7f626dbf8700 1 -- 172.21.15.67:0/769231158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6268104100 msgr2=0x7f626819fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:27.626 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.615+0000 7f626dbf8700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6268104100 0x7f626819fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.615+0000 7f626dbf8700 1 -- 172.21.15.67:0/769231158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6268108850 msgr2=0x7f6268199970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.615+0000 7f626dbf8700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6268108850 0x7f6268199970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.615+0000 7f626dbf8700 1 -- 172.21.15.67:0/769231158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626819a0d0 con 0x7f6268103750 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.615+0000 7f626e3f9700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6268108850 0x7f6268199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.615+0000 7f626dbf8700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268103750 0x7f626819f7b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f625c005910 tx=0x7f625c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.615+0000 7f625affd700 1 -- 172.21.15.67:0/769231158 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c026070 con 0x7f6268103750 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.615+0000 7f626fe5c700 1 -- 172.21.15.67:0/769231158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f626819a360 con 0x7f6268103750 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.615+0000 7f625affd700 1 -- 172.21.15.67:0/769231158 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f625c003c70 con 0x7f6268103750 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.615+0000 7f625affd700 1 -- 172.21.15.67:0/769231158 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f625c00fa40 con 0x7f6268103750 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.615+0000 7f626fe5c700 1 -- 172.21.15.67:0/769231158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62681a6680 con 0x7f6268103750 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.616+0000 7f626fe5c700 1 -- 172.21.15.67:0/769231158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6268104be0 con 0x7f6268103750 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.617+0000 7f625affd700 1 -- 172.21.15.67:0/769231158 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f625c029070 con 0x7f6268103750 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.618+0000 7f625affd700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f62540b2880 0x7f62540b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.618+0000 7f626d3f7700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f62540b2880 0x7f62540b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.618+0000 7f625affd700 1 -- 172.21.15.67:0/769231158 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f625c0e2ad0 con 0x7f6268103750 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.618+0000 7f626d3f7700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f62540b2880 0x7f62540b4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f62681030e0 tx=0x7f6264009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:27.627 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.621+0000 7f625affd700 1 -- 172.21.15.67:0/769231158 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f625c030080 con 0x7f6268103750 2024-05-03T17:20:27.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.882+0000 7f626fe5c700 1 -- 172.21.15.67:0/769231158 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f626802e510 con 0x7f6268103750 2024-05-03T17:20:27.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.882+0000 7f625affd700 1 -- 172.21.15.67:0/769231158 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7f625c013070 con 0x7f6268103750 2024-05-03T17:20:27.883 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:20:27.883 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2024-05-03T17:20:27.883 INFO:teuthology.orchestra.run.smithi067.stdout: { 2024-05-03T17:20:27.883 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2024-05-03T17:20:27.883 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2024-05-03T17:20:27.883 INFO:teuthology.orchestra.run.smithi067.stdout: } 2024-05-03T17:20:27.883 INFO:teuthology.orchestra.run.smithi067.stdout:] 2024-05-03T17:20:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.884+0000 7f626fe5c700 1 -- 172.21.15.67:0/769231158 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f62540b2880 msgr2=0x7f62540b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.884+0000 7f626fe5c700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f62540b2880 0x7f62540b4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f62681030e0 tx=0x7f6264009260 comp rx=0 tx=0).stop 2024-05-03T17:20:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.884+0000 7f626fe5c700 1 -- 172.21.15.67:0/769231158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268103750 msgr2=0x7f626819f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.884+0000 7f626fe5c700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268103750 0x7f626819f7b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f625c005910 tx=0x7f625c004270 comp rx=0 tx=0).stop 2024-05-03T17:20:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.884+0000 7f626fe5c700 1 -- 172.21.15.67:0/769231158 shutdown_connections 2024-05-03T17:20:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.884+0000 7f626fe5c700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f62540b2880 0x7f62540b4d40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.884+0000 7f626fe5c700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6268108850 0x7f6268199970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.884+0000 7f626fe5c700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6268103750 0x7f626819f7b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.884+0000 7f626fe5c700 1 --2- 172.21.15.67:0/769231158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6268104100 0x7f626819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:27.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.884+0000 7f626fe5c700 1 -- 172.21.15.67:0/769231158 >> 172.21.15.67:0/769231158 conn(0x7f626807a600 msgr2=0x7f6268101820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:27.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.884+0000 7f626fe5c700 1 -- 172.21.15.67:0/769231158 shutdown_connections 2024-05-03T17:20:27.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:27.884+0000 7f626fe5c700 1 -- 172.21.15.67:0/769231158 wait complete. 2024-05-03T17:20:28.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:27 smithi082 ceph-mon[33535]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:28.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:27 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/769231158' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-03T17:20:28.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:27 smithi070 ceph-mon[33250]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:28.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:27 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/769231158' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-03T17:20:28.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:27 smithi067 ceph-mon[29731]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:28.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:27 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/769231158' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-03T17:20:29.315 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-03T17:20:29.315 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-03T17:20:29.586 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:29.854 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:29 smithi067 ceph-mon[29731]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:29.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:29 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:29.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:29 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:29.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:29 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:29.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:29 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:30.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:29 smithi082 ceph-mon[33535]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:30.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:29 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:30.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:29 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:30.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:29 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:30.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:29 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:30.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:29 smithi070 ceph-mon[33250]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:30.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:29 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:30.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:29 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:30.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:29 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:30.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:29 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.321+0000 7f708601b700 1 -- 172.21.15.67:0/2417235160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080108480 msgr2=0x7f7080108860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.321+0000 7f708601b700 1 --2- 172.21.15.67:0/2417235160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080108480 0x7f7080108860 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7070009990 tx=0x7f7070009ca0 comp rx=0 tx=0).stop 2024-05-03T17:20:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.321+0000 7f708601b700 1 -- 172.21.15.67:0/2417235160 shutdown_connections 2024-05-03T17:20:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.321+0000 7f708601b700 1 --2- 172.21.15.67:0/2417235160 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7080102e90 0x7f708010f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.321+0000 7f708601b700 1 --2- 172.21.15.67:0/2417235160 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080108480 0x7f7080108860 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.321+0000 7f708601b700 1 --2- 172.21.15.67:0/2417235160 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f70801024d0 0x7f7080102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.321+0000 7f708601b700 1 -- 172.21.15.67:0/2417235160 >> 172.21.15.67:0/2417235160 conn(0x7f70800782b0 msgr2=0x7f70800786c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.321+0000 7f708601b700 1 -- 172.21.15.67:0/2417235160 shutdown_connections 2024-05-03T17:20:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.321+0000 7f708601b700 1 -- 172.21.15.67:0/2417235160 wait complete. 2024-05-03T17:20:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.321+0000 7f708601b700 1 Processor -- start 2024-05-03T17:20:31.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f708601b700 1 -- start start 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f708601b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f70801024d0 0x7f708019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f708601b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7080102e90 0x7f708019fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f708601b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080108480 0x7f7080199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f708601b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7080112590 con 0x7f7080108480 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f708601b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7080112410 con 0x7f7080102e90 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f708601b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7080112710 con 0x7f70801024d0 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f707ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080108480 0x7f7080199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f707f7fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f70801024d0 0x7f708019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f707ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080108480 0x7f7080199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58588/0 (socket says 172.21.15.67:58588) 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f707ffff700 1 -- 172.21.15.67:0/1344639865 learned_addr learned my addr 172.21.15.67:0/1344639865 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f707effd700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7080102e90 0x7f708019fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f707ffff700 1 -- 172.21.15.67:0/1344639865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f70801024d0 msgr2=0x7f708019f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f707ffff700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f70801024d0 0x7f708019f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f707ffff700 1 -- 172.21.15.67:0/1344639865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7080102e90 msgr2=0x7f708019fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:31.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f707ffff700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7080102e90 0x7f708019fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f707ffff700 1 -- 172.21.15.67:0/1344639865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f708019a220 con 0x7f7080108480 2024-05-03T17:20:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.322+0000 7f707f7fe700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f70801024d0 0x7f708019f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:20:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.323+0000 7f707ffff700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080108480 0x7f7080199ac0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7078007ee0 tx=0x7f707800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.323+0000 7f707cff9700 1 -- 172.21.15.67:0/1344639865 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7078004bb0 con 0x7f7080108480 2024-05-03T17:20:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.323+0000 7f708601b700 1 -- 172.21.15.67:0/1344639865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f708019a510 con 0x7f7080108480 2024-05-03T17:20:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.323+0000 7f707cff9700 1 -- 172.21.15.67:0/1344639865 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7078004d50 con 0x7f7080108480 2024-05-03T17:20:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.323+0000 7f707cff9700 1 -- 172.21.15.67:0/1344639865 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7078005920 con 0x7f7080108480 2024-05-03T17:20:31.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.323+0000 7f708601b700 1 -- 172.21.15.67:0/1344639865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70801a6780 con 0x7f7080108480 2024-05-03T17:20:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.324+0000 7f708601b700 1 -- 172.21.15.67:0/1344639865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f708006a190 con 0x7f7080108480 2024-05-03T17:20:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.325+0000 7f707cff9700 1 -- 172.21.15.67:0/1344639865 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f707801a020 con 0x7f7080108480 2024-05-03T17:20:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.326+0000 7f707cff9700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f70680b2880 0x7f70680b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.326+0000 7f707f7fe700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f70680b2880 0x7f70680b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.326+0000 7f707cff9700 1 -- 172.21.15.67:0/1344639865 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f70780e1a00 con 0x7f7080108480 2024-05-03T17:20:31.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.326+0000 7f707f7fe700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f70680b2880 0x7f70680b4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f707000b5c0 tx=0x7f7070005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:31.330 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.329+0000 7f707cff9700 1 -- 172.21.15.67:0/1344639865 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f7078010070 con 0x7f7080108480 2024-05-03T17:20:31.543 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:31.542+0000 7f708601b700 1 -- 172.21.15.67:0/1344639865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f708002e510 con 0x7f7080108480 2024-05-03T17:20:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:31 smithi067 ceph-mon[29731]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:20:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:31.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:31 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1344639865' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-03T17:20:32.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:31 smithi082 ceph-mon[33535]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:32.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:32.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:32.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:32.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:20:32.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:32.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:31 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1344639865' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-03T17:20:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:31 smithi070 ceph-mon[33250]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:20:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:20:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:20:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:31 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1344639865' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-03T17:20:32.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.148+0000 7f707cff9700 1 -- 172.21.15.67:0/1344639865 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7f70780a9bc0 con 0x7f7080108480 2024-05-03T17:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.151+0000 7f708601b700 1 -- 172.21.15.67:0/1344639865 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f70680b2880 msgr2=0x7f70680b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.151+0000 7f708601b700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f70680b2880 0x7f70680b4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f707000b5c0 tx=0x7f7070005960 comp rx=0 tx=0).stop 2024-05-03T17:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.151+0000 7f708601b700 1 -- 172.21.15.67:0/1344639865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080108480 msgr2=0x7f7080199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.151+0000 7f708601b700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080108480 0x7f7080199ac0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7078007ee0 tx=0x7f707800d3b0 comp rx=0 tx=0).stop 2024-05-03T17:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.151+0000 7f708601b700 1 -- 172.21.15.67:0/1344639865 shutdown_connections 2024-05-03T17:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.151+0000 7f708601b700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f70680b2880 0x7f70680b4d40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.151+0000 7f708601b700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7080108480 0x7f7080199ac0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.151+0000 7f708601b700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f70801024d0 0x7f708019f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.151+0000 7f708601b700 1 --2- 172.21.15.67:0/1344639865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7080102e90 0x7f708019fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:32.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.151+0000 7f708601b700 1 -- 172.21.15.67:0/1344639865 >> 172.21.15.67:0/1344639865 conn(0x7f70800782b0 msgr2=0x7f708006c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:32.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.151+0000 7f708601b700 1 -- 172.21.15.67:0/1344639865 shutdown_connections 2024-05-03T17:20:32.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:32.152+0000 7f708601b700 1 -- 172.21.15.67:0/1344639865 wait complete. 2024-05-03T17:20:32.157 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2024-05-03T17:20:32.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-03T17:20:33.112 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:33.416 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:33 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1344639865' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-03T17:20:33.416 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:33 smithi067 ceph-mon[29731]: osdmap e73: 12 total, 12 up, 12 in 2024-05-03T17:20:33.416 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:33 smithi067 ceph-mon[29731]: pgmap v286: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:33.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:33 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1344639865' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-03T17:20:33.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:33 smithi082 ceph-mon[33535]: osdmap e73: 12 total, 12 up, 12 in 2024-05-03T17:20:33.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:33 smithi082 ceph-mon[33535]: pgmap v286: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:33.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:33 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1344639865' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-03T17:20:33.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:33 smithi070 ceph-mon[33250]: osdmap e73: 12 total, 12 up, 12 in 2024-05-03T17:20:33.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:33 smithi070 ceph-mon[33250]: pgmap v286: 33 pgs: 1 creating+peering, 1 active+clean, 31 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:34.436 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:34 smithi067 ceph-mon[29731]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-03T17:20:34.436 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:34 smithi067 ceph-mon[29731]: osdmap e74: 12 total, 12 up, 12 in 2024-05-03T17:20:34.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:34 smithi082 ceph-mon[33535]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-03T17:20:34.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:34 smithi082 ceph-mon[33535]: osdmap e74: 12 total, 12 up, 12 in 2024-05-03T17:20:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:34 smithi070 ceph-mon[33250]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-03T17:20:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:34 smithi070 ceph-mon[33250]: osdmap e74: 12 total, 12 up, 12 in 2024-05-03T17:20:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.029+0000 7fa957b42700 1 -- 172.21.15.67:0/369128807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa950102d60 msgr2=0x7fa95010f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.029+0000 7fa957b42700 1 --2- 172.21.15.67:0/369128807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa950102d60 0x7fa95010f250 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa94800b0a0 tx=0x7fa94800b3b0 comp rx=0 tx=0).stop 2024-05-03T17:20:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.029+0000 7fa957b42700 1 -- 172.21.15.67:0/369128807 shutdown_connections 2024-05-03T17:20:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.029+0000 7fa957b42700 1 --2- 172.21.15.67:0/369128807 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa950102d60 0x7fa95010f250 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.029+0000 7fa957b42700 1 --2- 172.21.15.67:0/369128807 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa950108410 0x7fa9501087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.029+0000 7fa957b42700 1 --2- 172.21.15.67:0/369128807 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa9501023a0 0x7fa950102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.029+0000 7fa957b42700 1 -- 172.21.15.67:0/369128807 >> 172.21.15.67:0/369128807 conn(0x7fa950079350 msgr2=0x7fa950079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.029+0000 7fa957b42700 1 -- 172.21.15.67:0/369128807 shutdown_connections 2024-05-03T17:20:35.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.029+0000 7fa957b42700 1 -- 172.21.15.67:0/369128807 wait complete. 2024-05-03T17:20:35.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.029+0000 7fa957b42700 1 Processor -- start 2024-05-03T17:20:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa957b42700 1 -- start start 2024-05-03T17:20:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa957b42700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9501023a0 0x7fa950105f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa957b42700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa950102d60 0x7fa950104310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa957b42700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa950108410 0x7fa950104970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa957b42700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa950112560 con 0x7fa950108410 2024-05-03T17:20:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa957b42700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa9501123e0 con 0x7fa9501023a0 2024-05-03T17:20:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa957b42700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa9501126e0 con 0x7fa950102d60 2024-05-03T17:20:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa9560df700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa950108410 0x7fa950104970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa9560df700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa950108410 0x7fa950104970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:58608/0 (socket says 172.21.15.67:58608) 2024-05-03T17:20:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa9560df700 1 -- 172.21.15.67:0/835581423 learned_addr learned my addr 172.21.15.67:0/835581423 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa9558de700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9501023a0 0x7fa950105f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:35.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa9550dd700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa950102d60 0x7fa950104310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa9560df700 1 -- 172.21.15.67:0/835581423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa950102d60 msgr2=0x7fa950104310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa9560df700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa950102d60 0x7fa950104310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa9560df700 1 -- 172.21.15.67:0/835581423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9501023a0 msgr2=0x7fa950105f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa9560df700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9501023a0 0x7fa950105f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa9560df700 1 -- 172.21.15.67:0/835581423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9501ac550 con 0x7fa950108410 2024-05-03T17:20:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.030+0000 7fa9560df700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa950108410 0x7fa950104970 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa94800b830 tx=0x7fa948004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.031+0000 7fa946ffd700 1 -- 172.21.15.67:0/835581423 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa948021070 con 0x7fa950108410 2024-05-03T17:20:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.031+0000 7fa946ffd700 1 -- 172.21.15.67:0/835581423 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa948007e30 con 0x7fa950108410 2024-05-03T17:20:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.031+0000 7fa957b42700 1 -- 172.21.15.67:0/835581423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9501ac780 con 0x7fa950108410 2024-05-03T17:20:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.031+0000 7fa946ffd700 1 -- 172.21.15.67:0/835581423 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9480047d0 con 0x7fa950108410 2024-05-03T17:20:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.031+0000 7fa957b42700 1 -- 172.21.15.67:0/835581423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9501acbe0 con 0x7fa950108410 2024-05-03T17:20:35.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.032+0000 7fa957b42700 1 -- 172.21.15.67:0/835581423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa950103ac0 con 0x7fa950108410 2024-05-03T17:20:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.033+0000 7fa946ffd700 1 -- 172.21.15.67:0/835581423 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fa948007980 con 0x7fa950108410 2024-05-03T17:20:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.034+0000 7fa946ffd700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa93c0b2900 0x7fa93c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.034+0000 7fa9558de700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa93c0b2900 0x7fa93c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:35.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.034+0000 7fa946ffd700 1 -- 172.21.15.67:0/835581423 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7497+0+0 (secure 0 0 0) 0x7fa9480e2f20 con 0x7fa950108410 2024-05-03T17:20:35.037 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.035+0000 7fa9558de700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa93c0b2900 0x7fa93c0b4dc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa94c005950 tx=0x7fa94c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:35.040 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.039+0000 7fa946ffd700 1 -- 172.21.15.67:0/835581423 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fa9480ab000 con 0x7fa950108410 2024-05-03T17:20:35.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:35.255+0000 7fa957b42700 1 -- 172.21.15.67:0/835581423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fa9501acdb0 con 0x7fa950108410 2024-05-03T17:20:35.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:35 smithi067 ceph-mon[29731]: osdmap e75: 12 total, 12 up, 12 in 2024-05-03T17:20:35.299 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:35 smithi067 ceph-mon[29731]: pgmap v289: 33 pgs: 7 active+clean, 10 creating+peering, 16 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:35.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:35 smithi082 ceph-mon[33535]: osdmap e75: 12 total, 12 up, 12 in 2024-05-03T17:20:35.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:35 smithi082 ceph-mon[33535]: pgmap v289: 33 pgs: 7 active+clean, 10 creating+peering, 16 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:35.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:35 smithi070 ceph-mon[33250]: osdmap e75: 12 total, 12 up, 12 in 2024-05-03T17:20:35.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:35 smithi070 ceph-mon[33250]: pgmap v289: 33 pgs: 7 active+clean, 10 creating+peering, 16 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:36.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.172+0000 7fa946ffd700 1 -- 172.21.15.67:0/835581423 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v76) ==== 170+0+0 (secure 0 0 0) 0x7fa9480aa900 con 0x7fa950108410 2024-05-03T17:20:36.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.175+0000 7fa957b42700 1 -- 172.21.15.67:0/835581423 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa93c0b2900 msgr2=0x7fa93c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:36.176 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.175+0000 7fa957b42700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa93c0b2900 0x7fa93c0b4dc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa94c005950 tx=0x7fa94c00b410 comp rx=0 tx=0).stop 2024-05-03T17:20:36.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.175+0000 7fa957b42700 1 -- 172.21.15.67:0/835581423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa950108410 msgr2=0x7fa950104970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:36.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.175+0000 7fa957b42700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa950108410 0x7fa950104970 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa94800b830 tx=0x7fa948004ce0 comp rx=0 tx=0).stop 2024-05-03T17:20:36.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.176+0000 7fa957b42700 1 -- 172.21.15.67:0/835581423 shutdown_connections 2024-05-03T17:20:36.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.176+0000 7fa957b42700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa93c0b2900 0x7fa93c0b4dc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:36.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.176+0000 7fa957b42700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa950108410 0x7fa950104970 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:36.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.176+0000 7fa957b42700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa9501023a0 0x7fa950105f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:36.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.176+0000 7fa957b42700 1 --2- 172.21.15.67:0/835581423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa950102d60 0x7fa950104310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:36.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.176+0000 7fa957b42700 1 -- 172.21.15.67:0/835581423 >> 172.21.15.67:0/835581423 conn(0x7fa950079350 msgr2=0x7fa950106590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:36.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.176+0000 7fa957b42700 1 -- 172.21.15.67:0/835581423 shutdown_connections 2024-05-03T17:20:36.178 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:36.176+0000 7fa957b42700 1 -- 172.21.15.67:0/835581423 wait complete. 2024-05-03T17:20:36.179 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2024-05-03T17:20:36.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:36 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/835581423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-03T17:20:36.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:36 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/835581423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-03T17:20:36.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:36 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/835581423' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-03T17:20:37.053 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-03T17:20:37.323 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:37.445 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:37 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/835581423' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-03T17:20:37.445 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:37 smithi067 ceph-mon[29731]: osdmap e76: 12 total, 12 up, 12 in 2024-05-03T17:20:37.445 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:37 smithi067 ceph-mon[29731]: pgmap v291: 97 pgs: 18 active+clean, 14 creating+peering, 65 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:37.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:37 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/835581423' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-03T17:20:37.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:37 smithi082 ceph-mon[33535]: osdmap e76: 12 total, 12 up, 12 in 2024-05-03T17:20:37.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:37 smithi082 ceph-mon[33535]: pgmap v291: 97 pgs: 18 active+clean, 14 creating+peering, 65 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:37.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:37 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/835581423' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-03T17:20:37.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:37 smithi070 ceph-mon[33250]: osdmap e76: 12 total, 12 up, 12 in 2024-05-03T17:20:37.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:37 smithi070 ceph-mon[33250]: pgmap v291: 97 pgs: 18 active+clean, 14 creating+peering, 65 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:38.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:38 smithi067 ceph-mon[29731]: osdmap e77: 12 total, 12 up, 12 in 2024-05-03T17:20:38.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:38 smithi082 ceph-mon[33535]: osdmap e77: 12 total, 12 up, 12 in 2024-05-03T17:20:38.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:38 smithi070 ceph-mon[33250]: osdmap e77: 12 total, 12 up, 12 in 2024-05-03T17:20:39.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.046+0000 7fb1d051d700 1 -- 172.21.15.67:0/2518408815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c806c630 msgr2=0x7fb1c8110360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.046+0000 7fb1d051d700 1 --2- 172.21.15.67:0/2518408815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c806c630 0x7fb1c8110360 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb1c000b0d0 tx=0x7fb1c000b3e0 comp rx=0 tx=0).stop 2024-05-03T17:20:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.046+0000 7fb1d051d700 1 -- 172.21.15.67:0/2518408815 shutdown_connections 2024-05-03T17:20:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.046+0000 7fb1d051d700 1 --2- 172.21.15.67:0/2518408815 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c806c630 0x7fb1c8110360 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.046+0000 7fb1d051d700 1 --2- 172.21.15.67:0/2518408815 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1c8109590 0x7fb1c8109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.046+0000 7fb1d051d700 1 --2- 172.21.15.67:0/2518408815 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1c806bc70 0x7fb1c806c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.046+0000 7fb1d051d700 1 -- 172.21.15.67:0/2518408815 >> 172.21.15.67:0/2518408815 conn(0x7fb1c8079440 msgr2=0x7fb1c8079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.046+0000 7fb1d051d700 1 -- 172.21.15.67:0/2518408815 shutdown_connections 2024-05-03T17:20:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.046+0000 7fb1d051d700 1 -- 172.21.15.67:0/2518408815 wait complete. 2024-05-03T17:20:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.046+0000 7fb1d051d700 1 Processor -- start 2024-05-03T17:20:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.046+0000 7fb1d051d700 1 -- start start 2024-05-03T17:20:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1d051d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1c806bc70 0x7fb1c8102410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:39.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1d051d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c806c630 0x7fb1c8102950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1d051d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1c8109590 0x7fb1c8102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1d051d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c8113680 con 0x7fb1c806c630 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1d051d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c8113500 con 0x7fb1c8109590 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1d051d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c8113800 con 0x7fb1c806bc70 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1cdab8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c806c630 0x7fb1c8102950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1cdab8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c806c630 0x7fb1c8102950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54392/0 (socket says 172.21.15.67:54392) 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1cdab8700 1 -- 172.21.15.67:0/2096953970 learned_addr learned my addr 172.21.15.67:0/2096953970 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1ce2b9700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1c806bc70 0x7fb1c8102410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1ceaba700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1c8109590 0x7fb1c8102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1cdab8700 1 -- 172.21.15.67:0/2096953970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1c806bc70 msgr2=0x7fb1c8102410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1cdab8700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1c806bc70 0x7fb1c8102410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1cdab8700 1 -- 172.21.15.67:0/2096953970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1c8109590 msgr2=0x7fb1c8102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1cdab8700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1c8109590 0x7fb1c8102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1cdab8700 1 -- 172.21.15.67:0/2096953970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c81a8040 con 0x7fb1c806c630 2024-05-03T17:20:39.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.047+0000 7fb1cdab8700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c806c630 0x7fb1c8102950 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800d950 tx=0x7fb1b800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.048+0000 7fb1bf7fe700 1 -- 172.21.15.67:0/2096953970 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8016070 con 0x7fb1c806c630 2024-05-03T17:20:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.048+0000 7fb1d051d700 1 -- 172.21.15.67:0/2096953970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c81a8330 con 0x7fb1c806c630 2024-05-03T17:20:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.048+0000 7fb1bf7fe700 1 -- 172.21.15.67:0/2096953970 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb1b8009df0 con 0x7fb1c806c630 2024-05-03T17:20:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.048+0000 7fb1bf7fe700 1 -- 172.21.15.67:0/2096953970 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b800f3c0 con 0x7fb1c806c630 2024-05-03T17:20:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.048+0000 7fb1d051d700 1 -- 172.21.15.67:0/2096953970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c81a88c0 con 0x7fb1c806c630 2024-05-03T17:20:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.049+0000 7fb1d051d700 1 -- 172.21.15.67:0/2096953970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1c806a190 con 0x7fb1c806c630 2024-05-03T17:20:39.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.051+0000 7fb1bf7fe700 1 -- 172.21.15.67:0/2096953970 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fb1b8019070 con 0x7fb1c806c630 2024-05-03T17:20:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.052+0000 7fb1bf7fe700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb1b40b2880 0x7fb1b40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.052+0000 7fb1ce2b9700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb1b40b2880 0x7fb1b40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:39.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.052+0000 7fb1bf7fe700 1 -- 172.21.15.67:0/2096953970 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7879+0+0 (secure 0 0 0) 0x7fb1b80e1da0 con 0x7fb1c806c630 2024-05-03T17:20:39.054 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.052+0000 7fb1ce2b9700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb1b40b2880 0x7fb1b40b4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4005950 tx=0x7fb1c400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:39.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.055+0000 7fb1bf7fe700 1 -- 172.21.15.67:0/2096953970 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb1b80a9c70 con 0x7fb1c806c630 2024-05-03T17:20:39.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:39.320+0000 7fb1d051d700 1 -- 172.21.15.67:0/2096953970 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fb1c8103940 con 0x7fb1c806c630 2024-05-03T17:20:39.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:39 smithi067 ceph-mon[29731]: osdmap e78: 12 total, 12 up, 12 in 2024-05-03T17:20:39.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:39 smithi067 ceph-mon[29731]: pgmap v294: 97 pgs: 41 active+clean, 15 creating+peering, 41 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:39.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:39 smithi082 ceph-mon[33535]: osdmap e78: 12 total, 12 up, 12 in 2024-05-03T17:20:39.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:39 smithi082 ceph-mon[33535]: pgmap v294: 97 pgs: 41 active+clean, 15 creating+peering, 41 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:39.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:39 smithi070 ceph-mon[33250]: osdmap e78: 12 total, 12 up, 12 in 2024-05-03T17:20:39.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:39 smithi070 ceph-mon[33250]: pgmap v294: 97 pgs: 41 active+clean, 15 creating+peering, 41 unknown; 577 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:40.203 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.201+0000 7fb1bf7fe700 1 -- 172.21.15.67:0/2096953970 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fb1b80a9570 con 0x7fb1c806c630 2024-05-03T17:20:40.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.204+0000 7fb1d051d700 1 -- 172.21.15.67:0/2096953970 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb1b40b2880 msgr2=0x7fb1b40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:40.206 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.204+0000 7fb1d051d700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb1b40b2880 0x7fb1b40b4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4005950 tx=0x7fb1c400b410 comp rx=0 tx=0).stop 2024-05-03T17:20:40.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.204+0000 7fb1d051d700 1 -- 172.21.15.67:0/2096953970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c806c630 msgr2=0x7fb1c8102950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:40.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.204+0000 7fb1d051d700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c806c630 0x7fb1c8102950 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800d950 tx=0x7fb1b800dc60 comp rx=0 tx=0).stop 2024-05-03T17:20:40.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.204+0000 7fb1d051d700 1 -- 172.21.15.67:0/2096953970 shutdown_connections 2024-05-03T17:20:40.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.204+0000 7fb1d051d700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb1b40b2880 0x7fb1b40b4d40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:40.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.204+0000 7fb1d051d700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb1c8109590 0x7fb1c8102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:40.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.204+0000 7fb1d051d700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb1c806bc70 0x7fb1c8102410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:40.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.204+0000 7fb1d051d700 1 --2- 172.21.15.67:0/2096953970 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb1c806c630 0x7fb1c8102950 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:40.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.204+0000 7fb1d051d700 1 -- 172.21.15.67:0/2096953970 >> 172.21.15.67:0/2096953970 conn(0x7fb1c8079440 msgr2=0x7fb1c81010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:40.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.205+0000 7fb1d051d700 1 -- 172.21.15.67:0/2096953970 shutdown_connections 2024-05-03T17:20:40.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:40.205+0000 7fb1d051d700 1 -- 172.21.15.67:0/2096953970 wait complete. 2024-05-03T17:20:40.208 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 2 and data pool 3 2024-05-03T17:20:40.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:40 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2096953970' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-03T17:20:40.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:40 smithi067 ceph-mon[29731]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-03T17:20:40.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:40 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2096953970' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-03T17:20:40.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:40 smithi082 ceph-mon[33535]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-03T17:20:40.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:40 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2096953970' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-03T17:20:40.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:40 smithi070 ceph-mon[33250]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-03T17:20:41.032 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T17:20:41.294 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:41.416 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:41 smithi067 ceph-mon[29731]: osdmap e79: 12 total, 12 up, 12 in 2024-05-03T17:20:41.416 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:41 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2096953970' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-03T17:20:41.416 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:41 smithi067 ceph-mon[29731]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-03T17:20:41.416 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:41 smithi067 ceph-mon[29731]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-03T17:20:41.416 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:41 smithi067 ceph-mon[29731]: pgmap v296: 97 pgs: 74 active+clean, 15 creating+peering, 8 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:41.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:41 smithi082 ceph-mon[33535]: osdmap e79: 12 total, 12 up, 12 in 2024-05-03T17:20:41.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:41 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2096953970' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-03T17:20:41.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:41 smithi082 ceph-mon[33535]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-03T17:20:41.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:41 smithi082 ceph-mon[33535]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-03T17:20:41.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:41 smithi082 ceph-mon[33535]: pgmap v296: 97 pgs: 74 active+clean, 15 creating+peering, 8 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:41 smithi070 ceph-mon[33250]: osdmap e79: 12 total, 12 up, 12 in 2024-05-03T17:20:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:41 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2096953970' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-03T17:20:41.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:41 smithi070 ceph-mon[33250]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-03T17:20:41.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:41 smithi070 ceph-mon[33250]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-03T17:20:41.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:41 smithi070 ceph-mon[33250]: pgmap v296: 97 pgs: 74 active+clean, 15 creating+peering, 8 unknown; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:20:42.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:42 smithi067 ceph-mon[29731]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-03T17:20:42.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:42 smithi067 ceph-mon[29731]: Cluster is now healthy 2024-05-03T17:20:42.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:42 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] up:active 2024-05-03T17:20:42.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:42 smithi067 ceph-mon[29731]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T17:20:42.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:42 smithi067 ceph-mon[29731]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T17:20:42.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:42 smithi082 ceph-mon[33535]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-03T17:20:42.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:42 smithi082 ceph-mon[33535]: Cluster is now healthy 2024-05-03T17:20:42.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:42 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] up:active 2024-05-03T17:20:42.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:42 smithi082 ceph-mon[33535]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T17:20:42.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:42 smithi082 ceph-mon[33535]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T17:20:42.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:42 smithi070 ceph-mon[33250]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-03T17:20:42.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:42 smithi070 ceph-mon[33250]: Cluster is now healthy 2024-05-03T17:20:42.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:42 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] up:active 2024-05-03T17:20:42.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:42 smithi070 ceph-mon[33250]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T17:20:42.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:42 smithi070 ceph-mon[33250]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T17:20:43.278 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:43 smithi067 ceph-mon[29731]: pgmap v297: 97 pgs: 97 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 1 op/s 2024-05-03T17:20:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.281+0000 7f77f0277700 1 -- 172.21.15.67:0/2034304644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e81076c0 msgr2=0x7f77e810f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.281+0000 7f77f0277700 1 --2- 172.21.15.67:0/2034304644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e81076c0 0x7f77e810f290 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f77e000b0d0 tx=0x7f77e000b3e0 comp rx=0 tx=0).stop 2024-05-03T17:20:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.281+0000 7f77f0277700 1 -- 172.21.15.67:0/2034304644 shutdown_connections 2024-05-03T17:20:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.281+0000 7f77f0277700 1 --2- 172.21.15.67:0/2034304644 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e81076c0 0x7f77e810f290 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.281+0000 7f77f0277700 1 --2- 172.21.15.67:0/2034304644 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77e8102630 0x7f77e8102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.281+0000 7f77f0277700 1 --2- 172.21.15.67:0/2034304644 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77e8102fe0 0x7f77e8106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.281+0000 7f77f0277700 1 -- 172.21.15.67:0/2034304644 >> 172.21.15.67:0/2034304644 conn(0x7f77e8079350 msgr2=0x7f77e8079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.281+0000 7f77f0277700 1 -- 172.21.15.67:0/2034304644 shutdown_connections 2024-05-03T17:20:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.281+0000 7f77f0277700 1 -- 172.21.15.67:0/2034304644 wait complete. 2024-05-03T17:20:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.282+0000 7f77f0277700 1 Processor -- start 2024-05-03T17:20:43.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.282+0000 7f77f0277700 1 -- start start 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.282+0000 7f77f0277700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77e8102630 0x7f77e819a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.282+0000 7f77f0277700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e8102fe0 0x7f77e819aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.282+0000 7f77f0277700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77e81076c0 0x7f77e81a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.282+0000 7f77f0277700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f77e8112560 con 0x7f77e8102fe0 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.282+0000 7f77f0277700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77e81123e0 con 0x7f77e8102630 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.282+0000 7f77f0277700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f77e81126e0 con 0x7f77e81076c0 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.282+0000 7f77ed812700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e8102fe0 0x7f77e819aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.282+0000 7f77ed812700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e8102fe0 0x7f77e819aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54410/0 (socket says 172.21.15.67:54410) 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.282+0000 7f77ed812700 1 -- 172.21.15.67:0/3566570531 learned_addr learned my addr 172.21.15.67:0/3566570531 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77ee013700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77e8102630 0x7f77e819a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77ee814700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77e81076c0 0x7f77e81a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77ed812700 1 -- 172.21.15.67:0/3566570531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77e81076c0 msgr2=0x7f77e81a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77ed812700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77e81076c0 0x7f77e81a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77ed812700 1 -- 172.21.15.67:0/3566570531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77e8102630 msgr2=0x7f77e819a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:43.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77ed812700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77e8102630 0x7f77e819a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:43.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77ed812700 1 -- 172.21.15.67:0/3566570531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77e81a5f60 con 0x7f77e8102fe0 2024-05-03T17:20:43.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77ee814700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77e81076c0 0x7f77e81a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:20:43.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77ed812700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e8102fe0 0x7f77e819aee0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f77d800c8f0 tx=0x7f77d800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:43.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77df7fe700 1 -- 172.21.15.67:0/3566570531 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8007940 con 0x7f77e8102fe0 2024-05-03T17:20:43.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77df7fe700 1 -- 172.21.15.67:0/3566570531 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77d800f450 con 0x7f77e8102fe0 2024-05-03T17:20:43.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77f0277700 1 -- 172.21.15.67:0/3566570531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77e81a6190 con 0x7f77e8102fe0 2024-05-03T17:20:43.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77df7fe700 1 -- 172.21.15.67:0/3566570531 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8018660 con 0x7f77e8102fe0 2024-05-03T17:20:43.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.283+0000 7f77f0277700 1 -- 172.21.15.67:0/3566570531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77e81a66f0 con 0x7f77e8102fe0 2024-05-03T17:20:43.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.284+0000 7f77f0277700 1 -- 172.21.15.67:0/3566570531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77e806a190 con 0x7f77e8102fe0 2024-05-03T17:20:43.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.285+0000 7f77df7fe700 1 -- 172.21.15.67:0/3566570531 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f77d800f890 con 0x7f77e8102fe0 2024-05-03T17:20:43.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.286+0000 7f77df7fe700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f77d40b2900 0x7f77d40b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:43.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.286+0000 7f77ee013700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f77d40b2900 0x7f77d40b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:43.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.286+0000 7f77df7fe700 1 -- 172.21.15.67:0/3566570531 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f77d80e1ec0 con 0x7f77e8102fe0 2024-05-03T17:20:43.288 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.287+0000 7f77ee013700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f77d40b2900 0x7f77d40b4dc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f77e400ac50 tx=0x7f77e400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:43.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.291+0000 7f77df7fe700 1 -- 172.21.15.67:0/3566570531 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f77d8014030 con 0x7f77e8102fe0 2024-05-03T17:20:43.502 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:43 smithi070 ceph-mon[33250]: pgmap v297: 97 pgs: 97 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 1 op/s 2024-05-03T17:20:43.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.505+0000 7f77f0277700 1 -- 172.21.15.67:0/3566570531 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f77e802e630 con 0x7f77e8102fe0 2024-05-03T17:20:43.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.506+0000 7f77df7fe700 1 -- 172.21.15.67:0/3566570531 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22136 (secure 0 0 0) 0x7f77d80a9d30 con 0x7f77e8102fe0 2024-05-03T17:20:43.508 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:20:43.508 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":79,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","created":"2024-05-03T17:12:11.709381+0000","modified":"2024-05-03T17:20:39.322155+0000","last_up_change":"2024-05-03T17:19:29.310797+0000","last_in_change":"2024-05-03T17:19:08.848654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T17:15:36.413266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T17:20:31.546843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T17:20:35.261082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80962eb1-a506-49bb-a528-dee035eb4322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6803","nonce":1754365741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6805","nonce":1754365741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6809","nonce":1754365741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6807","nonce":1754365741}]},"public_addr":"172.21.15.67:6803/1754365741","cluster_addr":"172.21.15.67:6805/1754365741","heartbeat_back_addr":"172.21.15.67:6809/1754365741","heartbeat_front_addr":"172.21.15.67:6807/1754365741","state":["exists","up"]},{"osd":1,"uuid":"d7721d5f-4373-4cb2-bb57-bc368d6bb818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6801","nonce":1006339176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6803","nonce":1006339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6807","nonce":1006339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6805","nonce":1006339176}]},"public_addr":"172.21.15.70:6801/1006339176","cluster_addr":"172.21.15.70:6803/1006339176","heartbeat_back_addr":"172.21.15.70:6807/1006339176","heartbeat_front_addr":"172.21.15.70:6805/1006339176","state":["exists","up"]},{"osd":2,"uuid":"9992fd98-821b-40ce-b051-89befbfd87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6801","nonce":1888803264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6803","nonce":1888803264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6807","nonce":1888803264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6805","nonce":1888803264}]},"public_addr":"172.21.15.82:6801/1888803264","cluster_addr":"172.21.15.82:6803/1888803264","heartbeat_back_addr":"172.21.15.82:6807/1888803264","heartbeat_front_addr":"172.21.15.82:6805/1888803264","state":["exists","up"]},{"osd":3,"uuid":"14bfe6a6-0bc8-4766-8d36-fbaecc61a396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6811","nonce":313717723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6813","nonce":313717723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6817","nonce":313717723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6815","nonce":313717723}]},"public_addr":"172.21.15.67:6811/313717723","cluster_addr":"172.21.15.67:6813/313717723","heartbeat_back_addr":"172.21.15.67:6817/313717723","heartbeat_front_addr":"172.21.15.67:6815/313717723","state":["exists","up"]},{"osd":4,"uuid":"5a44aeb8-aabf-4824-be65-2c7170dbee52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6809","nonce":3526300828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6811","nonce":3526300828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6815","nonce":3526300828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6813","nonce":3526300828}]},"public_addr":"172.21.15.70:6809/3526300828","cluster_addr":"172.21.15.70:6811/3526300828","heartbeat_back_addr":"172.21.15.70:6815/3526300828","heartbeat_front_addr":"172.21.15.70:6813/3526300828","state":["exists","up"]},{"osd":5,"uuid":"a6ac56df-466e-4096-8fbf-835f5b544297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6809","nonce":3754223290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6811","nonce":3754223290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6815","nonce":3754223290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6813","nonce":3754223290}]},"public_addr":"172.21.15.82:6809/3754223290","cluster_addr":"172.21.15.82:6811/3754223290","heartbeat_back_addr":"172.21.15.82:6815/3754223290","heartbeat_front_addr":"172.21.15.82:6813/3754223290","state":["exists","up"]},{"osd":6,"uuid":"4d8a5e8a-66ac-462e-b960-24c0ee248fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6819","nonce":4033769161}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6821","nonce":4033769161}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6825","nonce":4033769161}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6823","nonce":4033769161}]},"public_addr":"172.21.15.67:6819/4033769161","cluster_addr":"172.21.15.67:6821/4033769161","heartbeat_back_addr":"172.21.15.67:6825/4033769161","heartbeat_front_addr":"172.21.15.67:6823/4033769161","state":["exists","up"]},{"osd":7,"uuid":"eb983837-8172-4802-9843-6360cda8f4be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6817","nonce":3366146173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6819","nonce":3366146173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6823","nonce":3366146173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6821","nonce":3366146173}]},"public_addr":"172.21.15.70:6817/3366146173","cluster_addr":"172.21.15.70:6819/3366146173","heartbeat_back_addr":"172.21.15.70:6823/3366146173","heartbeat_front_addr":"172.21.15.70:6821/3366146173","state":["exists","up"]},{"osd":8,"uuid":"c9d4ce56-e940-48d1-a9b5-437139e3fed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6817","nonce":347699286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6819","nonce":347699286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6823","nonce":347699286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6821","nonce":347699286}]},"public_addr":"172.21.15.82:6817/347699286","cluster_addr":"172.21.15.82:6819/347699286","heartbeat_back_addr":"172.21.15.82:6823/347699286","heartbeat_front_addr":"172.21.15.82:6821/347699286","state":["exists","up"]},{"osd":9,"uuid":"09596d19-f2d7-485e-a028-9ce5468c6651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6827","nonce":2600095198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6829","nonce":2600095198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6833","nonce":2600095198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6831","nonce":2600095198}]},"public_addr":"172.21.15.67:6827/2600095198","cluster_addr":"172.21.15.67:6829/2600095198","heartbeat_back_addr":"172.21.15.67:6833/2600095198","heartbeat_front_addr":"172.21.15.67:6831/2600095198","state":["exists","up"]},{"osd":10,"uuid":"ec4cffbc-0127-47e5-8176-0972f74a5514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6825","nonce":2897583629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6827","nonce":2897583629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6831","nonce":2897583629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6829","nonce":2897583629}]},"public_addr":"172.21.15.70:6825/2897583629","cluster_addr":"172.21.15.70:6827/2897583629","heartbeat_back_addr":"172.21.15.70:6831/2897583629","heartbeat_front_addr":"172.21.15.70:6829/2897583629","state":["exists","up"]},{"osd":11,"uuid":"a8acfb0b-98e8-49e1-a114-f33aa8160b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6825","nonce":4082656149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6827","nonce":4082656149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6831","nonce":4082656149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6829","nonce":4082656149}]},"public_addr":"172.21.15.82:6825/4082656149","cluster_addr":"172.21.15.82:6827/4082656149","heartbeat_back_addr":"172.21.15.82:6831/4082656149","heartbeat_front_addr":"172.21.15.82:6829/4082656149","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:14:45.596665+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:09.538408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:33.608945+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:59.803102+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:25.187966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:49.892160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:16.575973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:42.115955+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:07.655558+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:35.356779+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:01.144695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:27.577513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/2131289004":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/3335149382":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/992151181":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2614686067":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/3137329076":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2323686292":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/2579128777":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/3335149382":"2024-05-04T17:13:06.324002+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T17:20:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.509+0000 7f77f0277700 1 -- 172.21.15.67:0/3566570531 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f77d40b2900 msgr2=0x7f77d40b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.509+0000 7f77f0277700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f77d40b2900 0x7f77d40b4dc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f77e400ac50 tx=0x7f77e400a380 comp rx=0 tx=0).stop 2024-05-03T17:20:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.509+0000 7f77f0277700 1 -- 172.21.15.67:0/3566570531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e8102fe0 msgr2=0x7f77e819aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.509+0000 7f77f0277700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e8102fe0 0x7f77e819aee0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f77d800c8f0 tx=0x7f77d800cc00 comp rx=0 tx=0).stop 2024-05-03T17:20:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.509+0000 7f77f0277700 1 -- 172.21.15.67:0/3566570531 shutdown_connections 2024-05-03T17:20:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.509+0000 7f77f0277700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f77d40b2900 0x7f77d40b4dc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.509+0000 7f77f0277700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77e81076c0 0x7f77e81a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.509+0000 7f77f0277700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77e8102630 0x7f77e819a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.509+0000 7f77f0277700 1 --2- 172.21.15.67:0/3566570531 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77e8102fe0 0x7f77e819aee0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.509+0000 7f77f0277700 1 -- 172.21.15.67:0/3566570531 >> 172.21.15.67:0/3566570531 conn(0x7f77e8079350 msgr2=0x7f77e8100380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.509+0000 7f77f0277700 1 -- 172.21.15.67:0/3566570531 shutdown_connections 2024-05-03T17:20:43.511 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:43.509+0000 7f77f0277700 1 -- 172.21.15.67:0/3566570531 wait complete. 2024-05-03T17:20:43.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:43 smithi082 ceph-mon[33535]: pgmap v297: 97 pgs: 97 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 1 op/s 2024-05-03T17:20:43.808 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:20:43 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b[63855]: 2024-05-03T17:20:43.500+0000 7f03a9a7c700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-03T17:20:44.197 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T17:20:44.447 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:44.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:44 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3566570531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:20:44.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:44 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3566570531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:20:44.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:44 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3566570531' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:20:45.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:45 smithi082 ceph-mon[33535]: pgmap v298: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 3 op/s 2024-05-03T17:20:45.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:45 smithi070 ceph-mon[33250]: pgmap v298: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 3 op/s 2024-05-03T17:20:45.667 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:45 smithi067 ceph-mon[29731]: pgmap v298: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 3 op/s 2024-05-03T17:20:46.230 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.228+0000 7f5ce41b5700 1 -- 172.21.15.67:0/3937870344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc06bb40 msgr2=0x7f5cdc110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.228+0000 7f5ce41b5700 1 --2- 172.21.15.67:0/3937870344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc06bb40 0x7f5cdc110330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400b0a0 tx=0x7f5cd400b3b0 comp rx=0 tx=0).stop 2024-05-03T17:20:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.228+0000 7f5ce41b5700 1 -- 172.21.15.67:0/3937870344 shutdown_connections 2024-05-03T17:20:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.228+0000 7f5ce41b5700 1 --2- 172.21.15.67:0/3937870344 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc06bb40 0x7f5cdc110330 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.228+0000 7f5ce41b5700 1 --2- 172.21.15.67:0/3937870344 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cdc109510 0x7f5cdc1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.228+0000 7f5ce41b5700 1 --2- 172.21.15.67:0/3937870344 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cdc06b180 0x7f5cdc06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.228+0000 7f5ce41b5700 1 -- 172.21.15.67:0/3937870344 >> 172.21.15.67:0/3937870344 conn(0x7f5cdc079350 msgr2=0x7f5cdc079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.228+0000 7f5ce41b5700 1 -- 172.21.15.67:0/3937870344 shutdown_connections 2024-05-03T17:20:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.228+0000 7f5ce41b5700 1 -- 172.21.15.67:0/3937870344 wait complete. 2024-05-03T17:20:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.229+0000 7f5ce41b5700 1 Processor -- start 2024-05-03T17:20:46.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.229+0000 7f5ce41b5700 1 -- start start 2024-05-03T17:20:46.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.229+0000 7f5ce41b5700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cdc06b180 0x7f5cdc19f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:46.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.229+0000 7f5ce41b5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc06bb40 0x7f5cdc19fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:46.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.229+0000 7f5ce41b5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cdc109510 0x7f5cdc199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:46.233 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce41b5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc113620 con 0x7f5cdc06bb40 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce41b5700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1134a0 con 0x7f5cdc109510 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce41b5700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc1137a0 con 0x7f5cdc06b180 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce1750700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc06bb40 0x7f5cdc19fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce1750700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc06bb40 0x7f5cdc19fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:54432/0 (socket says 172.21.15.67:54432) 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce1750700 1 -- 172.21.15.67:0/1661229707 learned_addr learned my addr 172.21.15.67:0/1661229707 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce1f51700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cdc06b180 0x7f5cdc19f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce2752700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cdc109510 0x7f5cdc199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce1750700 1 -- 172.21.15.67:0/1661229707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cdc06b180 msgr2=0x7f5cdc19f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce1750700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cdc06b180 0x7f5cdc19f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce1750700 1 -- 172.21.15.67:0/1661229707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cdc109510 msgr2=0x7f5cdc199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce1750700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cdc109510 0x7f5cdc199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce1750700 1 -- 172.21.15.67:0/1661229707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cdc19a0c0 con 0x7f5cdc06bb40 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce2752700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cdc109510 0x7f5cdc199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:20:46.234 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce1750700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc06bb40 0x7f5cdc19fce0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00d950 tx=0x7f5ccc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:46.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5cd2ffd700 1 -- 172.21.15.67:0/1661229707 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc016070 con 0x7f5cdc06bb40 2024-05-03T17:20:46.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5cd2ffd700 1 -- 172.21.15.67:0/1661229707 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5ccc009df0 con 0x7f5cdc06bb40 2024-05-03T17:20:46.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.230+0000 7f5ce41b5700 1 -- 172.21.15.67:0/1661229707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cdc19a3b0 con 0x7f5cdc06bb40 2024-05-03T17:20:46.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.231+0000 7f5cd2ffd700 1 -- 172.21.15.67:0/1661229707 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc00f3c0 con 0x7f5cdc06bb40 2024-05-03T17:20:46.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.231+0000 7f5ce41b5700 1 -- 172.21.15.67:0/1661229707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cdc1023a0 con 0x7f5cdc06bb40 2024-05-03T17:20:46.235 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.232+0000 7f5ce41b5700 1 -- 172.21.15.67:0/1661229707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cdc06a190 con 0x7f5cdc06bb40 2024-05-03T17:20:46.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.233+0000 7f5cd2ffd700 1 -- 172.21.15.67:0/1661229707 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f5ccc019070 con 0x7f5cdc06bb40 2024-05-03T17:20:46.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.234+0000 7f5cd2ffd700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f5cc80b6c90 0x7f5cc80b9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:46.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.234+0000 7f5ce1f51700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f5cc80b6c90 0x7f5cc80b9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:46.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.234+0000 7f5cd2ffd700 1 -- 172.21.15.67:0/1661229707 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f5ccc0e1cd0 con 0x7f5cdc06bb40 2024-05-03T17:20:46.236 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.234+0000 7f5ce1f51700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f5cc80b6c90 0x7f5cc80b9150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc19aaa0 tx=0x7f5cd80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:46.238 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.237+0000 7f5cd2ffd700 1 -- 172.21.15.67:0/1661229707 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5ccc0a9b40 con 0x7f5cdc06bb40 2024-05-03T17:20:46.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.454+0000 7f5ce41b5700 1 -- 172.21.15.67:0/1661229707 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5cdc02e510 con 0x7f5cdc06bb40 2024-05-03T17:20:46.457 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.455+0000 7f5cd2ffd700 1 -- 172.21.15.67:0/1661229707 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22136 (secure 0 0 0) 0x7f5ccc0a9440 con 0x7f5cdc06bb40 2024-05-03T17:20:46.457 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:20:46.457 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":79,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","created":"2024-05-03T17:12:11.709381+0000","modified":"2024-05-03T17:20:39.322155+0000","last_up_change":"2024-05-03T17:19:29.310797+0000","last_in_change":"2024-05-03T17:19:08.848654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T17:15:36.413266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T17:20:31.546843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T17:20:35.261082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80962eb1-a506-49bb-a528-dee035eb4322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6803","nonce":1754365741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6805","nonce":1754365741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6809","nonce":1754365741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6807","nonce":1754365741}]},"public_addr":"172.21.15.67:6803/1754365741","cluster_addr":"172.21.15.67:6805/1754365741","heartbeat_back_addr":"172.21.15.67:6809/1754365741","heartbeat_front_addr":"172.21.15.67:6807/1754365741","state":["exists","up"]},{"osd":1,"uuid":"d7721d5f-4373-4cb2-bb57-bc368d6bb818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6801","nonce":1006339176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6803","nonce":1006339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6807","nonce":1006339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6805","nonce":1006339176}]},"public_addr":"172.21.15.70:6801/1006339176","cluster_addr":"172.21.15.70:6803/1006339176","heartbeat_back_addr":"172.21.15.70:6807/1006339176","heartbeat_front_addr":"172.21.15.70:6805/1006339176","state":["exists","up"]},{"osd":2,"uuid":"9992fd98-821b-40ce-b051-89befbfd87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6801","nonce":1888803264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6803","nonce":1888803264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6807","nonce":1888803264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6805","nonce":1888803264}]},"public_addr":"172.21.15.82:6801/1888803264","cluster_addr":"172.21.15.82:6803/1888803264","heartbeat_back_addr":"172.21.15.82:6807/1888803264","heartbeat_front_addr":"172.21.15.82:6805/1888803264","state":["exists","up"]},{"osd":3,"uuid":"14bfe6a6-0bc8-4766-8d36-fbaecc61a396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6811","nonce":313717723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6813","nonce":313717723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6817","nonce":313717723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6815","nonce":313717723}]},"public_addr":"172.21.15.67:6811/313717723","cluster_addr":"172.21.15.67:6813/313717723","heartbeat_back_addr":"172.21.15.67:6817/313717723","heartbeat_front_addr":"172.21.15.67:6815/313717723","state":["exists","up"]},{"osd":4,"uuid":"5a44aeb8-aabf-4824-be65-2c7170dbee52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6809","nonce":3526300828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6811","nonce":3526300828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6815","nonce":3526300828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6813","nonce":3526300828}]},"public_addr":"172.21.15.70:6809/3526300828","cluster_addr":"172.21.15.70:6811/3526300828","heartbeat_back_addr":"172.21.15.70:6815/3526300828","heartbeat_front_addr":"172.21.15.70:6813/3526300828","state":["exists","up"]},{"osd":5,"uuid":"a6ac56df-466e-4096-8fbf-835f5b544297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6809","nonce":3754223290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6811","nonce":3754223290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6815","nonce":3754223290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6813","nonce":3754223290}]},"public_addr":"172.21.15.82:6809/3754223290","cluster_addr":"172.21.15.82:6811/3754223290","heartbeat_back_addr":"172.21.15.82:6815/3754223290","heartbeat_front_addr":"172.21.15.82:6813/3754223290","state":["exists","up"]},{"osd":6,"uuid":"4d8a5e8a-66ac-462e-b960-24c0ee248fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6819","nonce":4033769161}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6821","nonce":4033769161}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6825","nonce":4033769161}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6823","nonce":4033769161}]},"public_addr":"172.21.15.67:6819/4033769161","cluster_addr":"172.21.15.67:6821/4033769161","heartbeat_back_addr":"172.21.15.67:6825/4033769161","heartbeat_front_addr":"172.21.15.67:6823/4033769161","state":["exists","up"]},{"osd":7,"uuid":"eb983837-8172-4802-9843-6360cda8f4be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6817","nonce":3366146173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6819","nonce":3366146173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6823","nonce":3366146173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6821","nonce":3366146173}]},"public_addr":"172.21.15.70:6817/3366146173","cluster_addr":"172.21.15.70:6819/3366146173","heartbeat_back_addr":"172.21.15.70:6823/3366146173","heartbeat_front_addr":"172.21.15.70:6821/3366146173","state":["exists","up"]},{"osd":8,"uuid":"c9d4ce56-e940-48d1-a9b5-437139e3fed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6817","nonce":347699286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6819","nonce":347699286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6823","nonce":347699286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6821","nonce":347699286}]},"public_addr":"172.21.15.82:6817/347699286","cluster_addr":"172.21.15.82:6819/347699286","heartbeat_back_addr":"172.21.15.82:6823/347699286","heartbeat_front_addr":"172.21.15.82:6821/347699286","state":["exists","up"]},{"osd":9,"uuid":"09596d19-f2d7-485e-a028-9ce5468c6651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6827","nonce":2600095198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6829","nonce":2600095198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6833","nonce":2600095198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6831","nonce":2600095198}]},"public_addr":"172.21.15.67:6827/2600095198","cluster_addr":"172.21.15.67:6829/2600095198","heartbeat_back_addr":"172.21.15.67:6833/2600095198","heartbeat_front_addr":"172.21.15.67:6831/2600095198","state":["exists","up"]},{"osd":10,"uuid":"ec4cffbc-0127-47e5-8176-0972f74a5514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6825","nonce":2897583629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6827","nonce":2897583629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6831","nonce":2897583629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6829","nonce":2897583629}]},"public_addr":"172.21.15.70:6825/2897583629","cluster_addr":"172.21.15.70:6827/2897583629","heartbeat_back_addr":"172.21.15.70:6831/2897583629","heartbeat_front_addr":"172.21.15.70:6829/2897583629","state":["exists","up"]},{"osd":11,"uuid":"a8acfb0b-98e8-49e1-a114-f33aa8160b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6825","nonce":4082656149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6827","nonce":4082656149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6831","nonce":4082656149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6829","nonce":4082656149}]},"public_addr":"172.21.15.82:6825/4082656149","cluster_addr":"172.21.15.82:6827/4082656149","heartbeat_back_addr":"172.21.15.82:6831/4082656149","heartbeat_front_addr":"172.21.15.82:6829/4082656149","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:14:45.596665+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:09.538408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:33.608945+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:59.803102+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:25.187966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:49.892160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:16.575973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:42.115955+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:07.655558+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:35.356779+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:01.144695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:27.577513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/2131289004":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/3335149382":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/992151181":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2614686067":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/3137329076":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2323686292":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/2579128777":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/3335149382":"2024-05-04T17:13:06.324002+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T17:20:46.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.457+0000 7f5ce41b5700 1 -- 172.21.15.67:0/1661229707 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f5cc80b6c90 msgr2=0x7f5cc80b9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:46.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.457+0000 7f5ce41b5700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f5cc80b6c90 0x7f5cc80b9150 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc19aaa0 tx=0x7f5cd80073d0 comp rx=0 tx=0).stop 2024-05-03T17:20:46.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.457+0000 7f5ce41b5700 1 -- 172.21.15.67:0/1661229707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc06bb40 msgr2=0x7f5cdc19fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:46.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.457+0000 7f5ce41b5700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc06bb40 0x7f5cdc19fce0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00d950 tx=0x7f5ccc00dc60 comp rx=0 tx=0).stop 2024-05-03T17:20:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.458+0000 7f5ce41b5700 1 -- 172.21.15.67:0/1661229707 shutdown_connections 2024-05-03T17:20:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.458+0000 7f5ce41b5700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f5cc80b6c90 0x7f5cc80b9150 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.458+0000 7f5ce41b5700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5cdc109510 0x7f5cdc199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.458+0000 7f5ce41b5700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5cdc06b180 0x7f5cdc19f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.458+0000 7f5ce41b5700 1 --2- 172.21.15.67:0/1661229707 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5cdc06bb40 0x7f5cdc19fce0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.458+0000 7f5ce41b5700 1 -- 172.21.15.67:0/1661229707 >> 172.21.15.67:0/1661229707 conn(0x7f5cdc079350 msgr2=0x7f5cdc101080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.458+0000 7f5ce41b5700 1 -- 172.21.15.67:0/1661229707 shutdown_connections 2024-05-03T17:20:46.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:46.458+0000 7f5ce41b5700 1 -- 172.21.15.67:0/1661229707 wait complete. 2024-05-03T17:20:46.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:46 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1661229707' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:20:46.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:46 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1661229707' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:20:46.842 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:46 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1661229707' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:20:47.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-05-03T17:20:47.552 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:47.678 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:47 smithi067 ceph-mon[29731]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-03T17:20:47.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:47 smithi082 ceph-mon[33535]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-03T17:20:47.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:47 smithi070 ceph-mon[33250]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-03T17:20:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.414+0000 7f8d20d18700 1 -- 172.21.15.67:0/2545784961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c108870 msgr2=0x7f8d1c1103f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.414+0000 7f8d20d18700 1 --2- 172.21.15.67:0/2545784961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c108870 0x7f8d1c1103f0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400b0d0 tx=0x7f8d1400b3e0 comp rx=0 tx=0).stop 2024-05-03T17:20:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.414+0000 7f8d20d18700 1 -- 172.21.15.67:0/2545784961 shutdown_connections 2024-05-03T17:20:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.414+0000 7f8d20d18700 1 --2- 172.21.15.67:0/2545784961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c108870 0x7f8d1c1103f0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.414+0000 7f8d20d18700 1 --2- 172.21.15.67:0/2545784961 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d1c103780 0x7f8d1c103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.414+0000 7f8d20d18700 1 --2- 172.21.15.67:0/2545784961 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d1c104130 0x7f8d1c108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.414+0000 7f8d20d18700 1 -- 172.21.15.67:0/2545784961 >> 172.21.15.67:0/2545784961 conn(0x7f8d1c07a610 msgr2=0x7f8d1c07aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.414+0000 7f8d20d18700 1 -- 172.21.15.67:0/2545784961 shutdown_connections 2024-05-03T17:20:49.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.414+0000 7f8d20d18700 1 -- 172.21.15.67:0/2545784961 wait complete. 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d20d18700 1 Processor -- start 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d20d18700 1 -- start start 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d20d18700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d1c103780 0x7f8d1c0757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d20d18700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d1c104130 0x7f8d1c075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d20d18700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c108870 0x7f8d1c076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d20d18700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c113690 con 0x7f8d1c108870 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d20d18700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c113510 con 0x7f8d1c104130 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d20d18700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c113810 con 0x7f8d1c103780 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d1ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c108870 0x7f8d1c076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d1ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c108870 0x7f8d1c076220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41254/0 (socket says 172.21.15.67:41254) 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d1ad9d700 1 -- 172.21.15.67:0/872818628 learned_addr learned my addr 172.21.15.67:0/872818628 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d1a59c700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d1c103780 0x7f8d1c0757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:49.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d19d9b700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d1c104130 0x7f8d1c075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d1ad9d700 1 -- 172.21.15.67:0/872818628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d1c103780 msgr2=0x7f8d1c0757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d1ad9d700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d1c103780 0x7f8d1c0757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d1ad9d700 1 -- 172.21.15.67:0/872818628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d1c104130 msgr2=0x7f8d1c075ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d1ad9d700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d1c104130 0x7f8d1c075ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d1ad9d700 1 -- 172.21.15.67:0/872818628 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1c1a7f70 con 0x7f8d1c108870 2024-05-03T17:20:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.415+0000 7f8d19d9b700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d1c104130 0x7f8d1c075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:20:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.416+0000 7f8d1ad9d700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c108870 0x7f8d1c076220 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400d010 tx=0x7f8d140079a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.416+0000 7f8d0b7fe700 1 -- 172.21.15.67:0/872818628 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1401f070 con 0x7f8d1c108870 2024-05-03T17:20:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.416+0000 7f8d20d18700 1 -- 172.21.15.67:0/872818628 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1c1a8200 con 0x7f8d1c108870 2024-05-03T17:20:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.416+0000 7f8d0b7fe700 1 -- 172.21.15.67:0/872818628 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d14003c00 con 0x7f8d1c108870 2024-05-03T17:20:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.416+0000 7f8d0b7fe700 1 -- 172.21.15.67:0/872818628 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1401b6a0 con 0x7f8d1c108870 2024-05-03T17:20:49.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.416+0000 7f8d20d18700 1 -- 172.21.15.67:0/872818628 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d1c1a8660 con 0x7f8d1c108870 2024-05-03T17:20:49.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.417+0000 7f8d20d18700 1 -- 172.21.15.67:0/872818628 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d1c06a190 con 0x7f8d1c108870 2024-05-03T17:20:49.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.419+0000 7f8d0b7fe700 1 -- 172.21.15.67:0/872818628 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f8d14022070 con 0x7f8d1c108870 2024-05-03T17:20:49.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.419+0000 7f8d0b7fe700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8d040b2880 0x7f8d040b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.420+0000 7f8d1a59c700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8d040b2880 0x7f8d040b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.420+0000 7f8d0b7fe700 1 -- 172.21.15.67:0/872818628 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f8d1400f020 con 0x7f8d1c108870 2024-05-03T17:20:49.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.420+0000 7f8d1a59c700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8d040b2880 0x7f8d040b4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00ac50 tx=0x7f8d0c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:49.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.423+0000 7f8d0b7fe700 1 -- 172.21.15.67:0/872818628 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f8d14029020 con 0x7f8d1c108870 2024-05-03T17:20:49.685 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:49.685+0000 7f8d20d18700 1 -- 172.21.15.67:0/872818628 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f8d1c1a89c0 con 0x7f8d1c108870 2024-05-03T17:20:49.686 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:49 smithi067 ceph-mon[29731]: pgmap v300: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-03T17:20:49.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:49 smithi082 ceph-mon[33535]: pgmap v300: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-03T17:20:49.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:49 smithi070 ceph-mon[33250]: pgmap v300: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-03T17:20:50.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.520+0000 7f8d0b7fe700 1 -- 172.21.15.67:0/872818628 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f8d140ab500 con 0x7f8d1c108870 2024-05-03T17:20:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.523+0000 7f8d20d18700 1 -- 172.21.15.67:0/872818628 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8d040b2880 msgr2=0x7f8d040b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.523+0000 7f8d20d18700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8d040b2880 0x7f8d040b4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c00ac50 tx=0x7f8d0c00a380 comp rx=0 tx=0).stop 2024-05-03T17:20:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.523+0000 7f8d20d18700 1 -- 172.21.15.67:0/872818628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c108870 msgr2=0x7f8d1c076220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.523+0000 7f8d20d18700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c108870 0x7f8d1c076220 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400d010 tx=0x7f8d140079a0 comp rx=0 tx=0).stop 2024-05-03T17:20:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.523+0000 7f8d20d18700 1 -- 172.21.15.67:0/872818628 shutdown_connections 2024-05-03T17:20:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.523+0000 7f8d20d18700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8d040b2880 0x7f8d040b4d40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.523+0000 7f8d20d18700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8d1c108870 0x7f8d1c076220 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:50.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.524+0000 7f8d20d18700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8d1c103780 0x7f8d1c0757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.524+0000 7f8d20d18700 1 --2- 172.21.15.67:0/872818628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8d1c104130 0x7f8d1c075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.524+0000 7f8d20d18700 1 -- 172.21.15.67:0/872818628 >> 172.21.15.67:0/872818628 conn(0x7f8d1c07a610 msgr2=0x7f8d1c101490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.524+0000 7f8d20d18700 1 -- 172.21.15.67:0/872818628 shutdown_connections 2024-05-03T17:20:50.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:50.524+0000 7f8d20d18700 1 -- 172.21.15.67:0/872818628 wait complete. 2024-05-03T17:20:50.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:50 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/872818628' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-03T17:20:50.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:50 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/872818628' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-03T17:20:50.904 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:50 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/872818628' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-03T17:20:51.373 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-05-03T17:20:51.373 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs set cephfs max_mds 3 2024-05-03T17:20:51.628 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:51.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:51 smithi067 ceph-mon[29731]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-03T17:20:51.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:51 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/872818628' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-03T17:20:51.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:51 smithi067 ceph-mon[29731]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T17:20:51.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:51 smithi070 ceph-mon[33250]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-03T17:20:51.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:51 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/872818628' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-03T17:20:51.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:51 smithi070 ceph-mon[33250]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T17:20:51.940 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:51 smithi082 ceph-mon[33535]: pgmap v301: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-03T17:20:51.941 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:51 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/872818628' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-03T17:20:51.941 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:51 smithi082 ceph-mon[33535]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T17:20:53.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.343+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/2860663125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3abc1023a0 msgr2=0x7f3abc102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:53.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.343+0000 7f3ac3d4e700 1 --2- 172.21.15.67:0/2860663125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3abc1023a0 0x7f3abc102820 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3aac009990 tx=0x7f3aac009ca0 comp rx=0 tx=0).stop 2024-05-03T17:20:53.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.343+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/2860663125 shutdown_connections 2024-05-03T17:20:53.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.343+0000 7f3ac3d4e700 1 --2- 172.21.15.67:0/2860663125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3abc102d60 0x7f3abc10f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:53.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.343+0000 7f3ac3d4e700 1 --2- 172.21.15.67:0/2860663125 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3abc108410 0x7f3abc1087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:53.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.343+0000 7f3ac3d4e700 1 --2- 172.21.15.67:0/2860663125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3abc1023a0 0x7f3abc102820 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.343+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/2860663125 >> 172.21.15.67:0/2860663125 conn(0x7f3abc079350 msgr2=0x7f3abc079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.343+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/2860663125 shutdown_connections 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.343+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/2860663125 wait complete. 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.344+0000 7f3ac3d4e700 1 Processor -- start 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.344+0000 7f3ac3d4e700 1 -- start start 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.344+0000 7f3ac3d4e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3abc1023a0 0x7f3abc19e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.344+0000 7f3ac3d4e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3abc102d60 0x7f3abc19ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.344+0000 7f3ac3d4e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3abc108410 0x7f3abc198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.344+0000 7f3ac3d4e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc112680 con 0x7f3abc102d60 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.344+0000 7f3ac3d4e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc112500 con 0x7f3abc108410 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.344+0000 7f3ac3d4e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3abc112800 con 0x7f3abc1023a0 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac12e9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3abc102d60 0x7f3abc19ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:53.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac1aea700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3abc1023a0 0x7f3abc19e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac12e9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3abc102d60 0x7f3abc19ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:41280/0 (socket says 172.21.15.67:41280) 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac12e9700 1 -- 172.21.15.67:0/1982118342 learned_addr learned my addr 172.21.15.67:0/1982118342 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac22eb700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3abc108410 0x7f3abc198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac12e9700 1 -- 172.21.15.67:0/1982118342 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3abc1023a0 msgr2=0x7f3abc19e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac12e9700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3abc1023a0 0x7f3abc19e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac12e9700 1 -- 172.21.15.67:0/1982118342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3abc108410 msgr2=0x7f3abc198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac12e9700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3abc108410 0x7f3abc198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac12e9700 1 -- 172.21.15.67:0/1982118342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3abc199240 con 0x7f3abc102d60 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac1aea700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3abc1023a0 0x7f3abc19e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac22eb700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3abc108410 0x7f3abc198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac12e9700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3abc102d60 0x7f3abc19ee60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3aac005910 tx=0x7f3aac0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ab2ffd700 1 -- 172.21.15.67:0/1982118342 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aac027070 con 0x7f3abc102d60 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ab2ffd700 1 -- 172.21.15.67:0/1982118342 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aac004020 con 0x7f3abc102d60 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/1982118342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3abc1994d0 con 0x7f3abc102d60 2024-05-03T17:20:53.353 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ab2ffd700 1 -- 172.21.15.67:0/1982118342 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aac022e00 con 0x7f3abc102d60 2024-05-03T17:20:53.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.345+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/1982118342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3abc1a57f0 con 0x7f3abc102d60 2024-05-03T17:20:53.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.346+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/1982118342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3abc10a980 con 0x7f3abc102d60 2024-05-03T17:20:53.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.348+0000 7f3ab2ffd700 1 -- 172.21.15.67:0/1982118342 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f3aac02a070 con 0x7f3abc102d60 2024-05-03T17:20:53.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.348+0000 7f3ab2ffd700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f3aa80b2900 0x7f3aa80b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:53.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.349+0000 7f3ab2ffd700 1 -- 172.21.15.67:0/1982118342 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f3aac0e3660 con 0x7f3abc102d60 2024-05-03T17:20:53.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.349+0000 7f3ac1aea700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f3aa80b2900 0x7f3aa80b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:53.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.349+0000 7f3ac1aea700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f3aa80b2900 0x7f3aa80b4dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8009de0 tx=0x7f3ab8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:53.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.352+0000 7f3ab2ffd700 1 -- 172.21.15.67:0/1982118342 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f3aac031070 con 0x7f3abc102d60 2024-05-03T17:20:53.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:53.615+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/1982118342 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f3abc199fd0 con 0x7f3abc102d60 2024-05-03T17:20:53.616 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:53 smithi067 ceph-mon[29731]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-05-03T17:20:53.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:53 smithi070 ceph-mon[33250]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-05-03T17:20:54.018 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:53 smithi082 ceph-mon[33535]: pgmap v302: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-05-03T17:20:54.535 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.535+0000 7f3ab2ffd700 1 -- 172.21.15.67:0/1982118342 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f3aac013070 con 0x7f3abc102d60 2024-05-03T17:20:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.538+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/1982118342 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f3aa80b2900 msgr2=0x7f3aa80b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.538+0000 7f3ac3d4e700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f3aa80b2900 0x7f3aa80b4dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8009de0 tx=0x7f3ab8009190 comp rx=0 tx=0).stop 2024-05-03T17:20:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.538+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/1982118342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3abc102d60 msgr2=0x7f3abc19ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.538+0000 7f3ac3d4e700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3abc102d60 0x7f3abc19ee60 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3aac005910 tx=0x7f3aac0047b0 comp rx=0 tx=0).stop 2024-05-03T17:20:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.538+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/1982118342 shutdown_connections 2024-05-03T17:20:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.538+0000 7f3ac3d4e700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f3aa80b2900 0x7f3aa80b4dc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.538+0000 7f3ac3d4e700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3abc108410 0x7f3abc198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:54.539 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.538+0000 7f3ac3d4e700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3abc1023a0 0x7f3abc19e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:54.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.538+0000 7f3ac3d4e700 1 --2- 172.21.15.67:0/1982118342 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3abc102d60 0x7f3abc19ee60 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:54.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.538+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/1982118342 >> 172.21.15.67:0/1982118342 conn(0x7f3abc079350 msgr2=0x7f3abc101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:54.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.538+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/1982118342 shutdown_connections 2024-05-03T17:20:54.540 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:54.538+0000 7f3ac3d4e700 1 -- 172.21.15.67:0/1982118342 wait complete. 2024-05-03T17:20:54.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:54 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1982118342' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-03T17:20:54.922 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:54 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1982118342' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-03T17:20:55.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:54 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1982118342' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-03T17:20:55.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-05-03T17:20:55.599 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:55.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:55 smithi067 ceph-mon[29731]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 3 op/s 2024-05-03T17:20:55.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:55 smithi067 ceph-mon[29731]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-03T17:20:55.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:55 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1982118342' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-03T17:20:55.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:55 smithi067 ceph-mon[29731]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-03T17:20:55.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:55 smithi067 ceph-mon[29731]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T17:20:55.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:55 smithi067 ceph-mon[29731]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-03T17:20:55.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:55 smithi067 ceph-mon[29731]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-03T17:20:55.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:55 smithi070 ceph-mon[33250]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 3 op/s 2024-05-03T17:20:55.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:55 smithi070 ceph-mon[33250]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-03T17:20:55.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:55 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1982118342' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-03T17:20:55.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:55 smithi070 ceph-mon[33250]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-03T17:20:55.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:55 smithi070 ceph-mon[33250]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T17:20:55.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:55 smithi070 ceph-mon[33250]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-03T17:20:55.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:55 smithi070 ceph-mon[33250]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-03T17:20:56.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:55 smithi082 ceph-mon[33535]: pgmap v303: 97 pgs: 97 active+clean; 579 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 3 op/s 2024-05-03T17:20:56.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:55 smithi082 ceph-mon[33535]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-03T17:20:56.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:55 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1982118342' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-03T17:20:56.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:55 smithi082 ceph-mon[33535]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-03T17:20:56.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:55 smithi082 ceph-mon[33535]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-03T17:20:56.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:55 smithi082 ceph-mon[33535]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-03T17:20:56.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:55 smithi082 ceph-mon[33535]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-03T17:20:56.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:56 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] up:active 2024-05-03T17:20:56.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:56 smithi067 ceph-mon[29731]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-03T17:20:56.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:56 smithi067 ceph-mon[29731]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-03T17:20:56.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:56 smithi067 ceph-mon[29731]: Cluster is now healthy 2024-05-03T17:20:56.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:56 smithi067 ceph-mon[29731]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-03T17:20:56.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:56 smithi067 ceph-mon[29731]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-05-03T17:20:56.704 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:56 smithi067 ceph-mon[29731]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-05-03T17:20:56.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:56 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] up:active 2024-05-03T17:20:56.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:56 smithi070 ceph-mon[33250]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-03T17:20:56.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:56 smithi070 ceph-mon[33250]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-03T17:20:56.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:56 smithi070 ceph-mon[33250]: Cluster is now healthy 2024-05-03T17:20:56.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:56 smithi070 ceph-mon[33250]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-03T17:20:56.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:56 smithi070 ceph-mon[33250]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-05-03T17:20:56.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:56 smithi070 ceph-mon[33250]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-05-03T17:20:57.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:56 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] up:active 2024-05-03T17:20:57.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:56 smithi082 ceph-mon[33535]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-03T17:20:57.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:56 smithi082 ceph-mon[33535]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-03T17:20:57.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:56 smithi082 ceph-mon[33535]: Cluster is now healthy 2024-05-03T17:20:57.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:56 smithi082 ceph-mon[33535]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-03T17:20:57.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:56 smithi082 ceph-mon[33535]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-05-03T17:20:57.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:56 smithi082 ceph-mon[33535]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-05-03T17:20:57.304 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.304+0000 7f075c4c6700 1 -- 172.21.15.67:0/3195304814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0754102620 msgr2=0x7f0754102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:57.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.304+0000 7f075c4c6700 1 --2- 172.21.15.67:0/3195304814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0754102620 0x7f0754102a00 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07500098f0 tx=0x7f0750009c00 comp rx=0 tx=0).stop 2024-05-03T17:20:57.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.304+0000 7f075c4c6700 1 -- 172.21.15.67:0/3195304814 shutdown_connections 2024-05-03T17:20:57.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.304+0000 7f075c4c6700 1 --2- 172.21.15.67:0/3195304814 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0754107700 0x7f075410f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:57.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.304+0000 7f075c4c6700 1 --2- 172.21.15.67:0/3195304814 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0754102620 0x7f0754102a00 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:57.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.304+0000 7f075c4c6700 1 --2- 172.21.15.67:0/3195304814 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0754102fd0 0x7f0754107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:57.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.304+0000 7f075c4c6700 1 -- 172.21.15.67:0/3195304814 >> 172.21.15.67:0/3195304814 conn(0x7f0754079440 msgr2=0x7f0754079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:57.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.304+0000 7f075c4c6700 1 -- 172.21.15.67:0/3195304814 shutdown_connections 2024-05-03T17:20:57.305 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.304+0000 7f075c4c6700 1 -- 172.21.15.67:0/3195304814 wait complete. 2024-05-03T17:20:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.304+0000 7f075c4c6700 1 Processor -- start 2024-05-03T17:20:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f075c4c6700 1 -- start start 2024-05-03T17:20:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f075c4c6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0754102620 0x7f075419f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f075c4c6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0754102fd0 0x7f075419fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f075c4c6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0754107700 0x7f0754199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f075c4c6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0754112590 con 0x7f0754102fd0 2024-05-03T17:20:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f075c4c6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0754112410 con 0x7f0754102620 2024-05-03T17:20:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f075c4c6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0754112710 con 0x7f0754107700 2024-05-03T17:20:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f075a262700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0754102620 0x7f075419f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f0759a61700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0754102fd0 0x7f075419fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f075a262700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0754102620 0x7f075419f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.67:56008/0 (socket says 172.21.15.67:56008) 2024-05-03T17:20:57.307 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f075a262700 1 -- 172.21.15.67:0/2490503121 learned_addr learned my addr 172.21.15.67:0/2490503121 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f075aa63700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0754107700 0x7f0754199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f0759a61700 1 -- 172.21.15.67:0/2490503121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0754107700 msgr2=0x7f0754199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f0759a61700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0754107700 0x7f0754199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f0759a61700 1 -- 172.21.15.67:0/2490503121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0754102620 msgr2=0x7f075419f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f0759a61700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0754102620 0x7f075419f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f0759a61700 1 -- 172.21.15.67:0/2490503121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f075419a1d0 con 0x7f0754102fd0 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f075a262700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0754102620 0x7f075419f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.305+0000 7f0759a61700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0754102fd0 0x7f075419fda0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f075406ba30 tx=0x7f074400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.306+0000 7f074b7fe700 1 -- 172.21.15.67:0/2490503121 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0744017070 con 0x7f0754102fd0 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.306+0000 7f075c4c6700 1 -- 172.21.15.67:0/2490503121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f075419a4c0 con 0x7f0754102fd0 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.306+0000 7f074b7fe700 1 -- 172.21.15.67:0/2490503121 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0744009de0 con 0x7f0754102fd0 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.306+0000 7f074b7fe700 1 -- 172.21.15.67:0/2490503121 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0744010d90 con 0x7f0754102fd0 2024-05-03T17:20:57.308 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.306+0000 7f075c4c6700 1 -- 172.21.15.67:0/2490503121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07541a6730 con 0x7f0754102fd0 2024-05-03T17:20:57.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.307+0000 7f075c4c6700 1 -- 172.21.15.67:0/2490503121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f075406a190 con 0x7f0754102fd0 2024-05-03T17:20:57.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.308+0000 7f074b7fe700 1 -- 172.21.15.67:0/2490503121 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7f074401a070 con 0x7f0754102fd0 2024-05-03T17:20:57.309 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.309+0000 7f074b7fe700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f07400b2880 0x7f07400b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:20:57.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.309+0000 7f075a262700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f07400b2880 0x7f07400b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:20:57.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.309+0000 7f074b7fe700 1 -- 172.21.15.67:0/2490503121 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f07440e19d0 con 0x7f0754102fd0 2024-05-03T17:20:57.310 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.309+0000 7f075a262700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f07400b2880 0x7f07400b4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0750009680 tx=0x7f0750019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:20:57.313 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.312+0000 7f074b7fe700 1 -- 172.21.15.67:0/2490503121 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f07440a9840 con 0x7f0754102fd0 2024-05-03T17:20:57.579 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:57.578+0000 7f075c4c6700 1 -- 172.21.15.67:0/2490503121 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f075419af50 con 0x7f0754102fd0 2024-05-03T17:20:57.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:57 smithi070 ceph-mon[33250]: pgmap v304: 97 pgs: 97 active+clean; 580 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 3 op/s 2024-05-03T17:20:57.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:57 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] up:active 2024-05-03T17:20:57.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:57 smithi070 ceph-mon[33250]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-05-03T17:20:57.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:57 smithi067 ceph-mon[29731]: pgmap v304: 97 pgs: 97 active+clean; 580 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 3 op/s 2024-05-03T17:20:57.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:57 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] up:active 2024-05-03T17:20:57.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:57 smithi067 ceph-mon[29731]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-05-03T17:20:58.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:57 smithi082 ceph-mon[33535]: pgmap v304: 97 pgs: 97 active+clean; 580 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 3 op/s 2024-05-03T17:20:58.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:57 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] up:active 2024-05-03T17:20:58.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:57 smithi082 ceph-mon[33535]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-05-03T17:20:58.563 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.563+0000 7f074b7fe700 1 -- 172.21.15.67:0/2490503121 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v23) ==== 121+0+0 (secure 0 0 0) 0x7f075419af50 con 0x7f0754102fd0 2024-05-03T17:20:58.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.566+0000 7f075c4c6700 1 -- 172.21.15.67:0/2490503121 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f07400b2880 msgr2=0x7f07400b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.566+0000 7f075c4c6700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f07400b2880 0x7f07400b4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0750009680 tx=0x7f0750019040 comp rx=0 tx=0).stop 2024-05-03T17:20:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.566+0000 7f075c4c6700 1 -- 172.21.15.67:0/2490503121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0754102fd0 msgr2=0x7f075419fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:20:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.566+0000 7f075c4c6700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0754102fd0 0x7f075419fda0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f075406ba30 tx=0x7f074400dc30 comp rx=0 tx=0).stop 2024-05-03T17:20:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.566+0000 7f075c4c6700 1 -- 172.21.15.67:0/2490503121 shutdown_connections 2024-05-03T17:20:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.566+0000 7f075c4c6700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f07400b2880 0x7f07400b4d40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.566+0000 7f075c4c6700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0754107700 0x7f0754199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.566+0000 7f075c4c6700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0754102620 0x7f075419f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.566+0000 7f075c4c6700 1 --2- 172.21.15.67:0/2490503121 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0754102fd0 0x7f075419fda0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:20:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.566+0000 7f075c4c6700 1 -- 172.21.15.67:0/2490503121 >> 172.21.15.67:0/2490503121 conn(0x7f0754079440 msgr2=0x7f0754106000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:20:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.566+0000 7f075c4c6700 1 -- 172.21.15.67:0/2490503121 shutdown_connections 2024-05-03T17:20:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:20:58.566+0000 7f075c4c6700 1 -- 172.21.15.67:0/2490503121 wait complete. 2024-05-03T17:20:58.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:58 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2490503121' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-03T17:20:58.808 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:20:58 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b[63855]: 2024-05-03T17:20:58.502+0000 7f03a9a7c700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-05-03T17:20:58.899 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:58 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2490503121' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-03T17:20:59.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:58 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2490503121' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-03T17:20:59.346 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-05-03T17:20:59.601 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:20:59.880 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:59 smithi067 ceph-mon[29731]: pgmap v305: 97 pgs: 97 active+clean; 581 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-05-03T17:20:59.880 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:59 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2490503121' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-03T17:20:59.880 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:59 smithi067 ceph-mon[29731]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-05-03T17:20:59.880 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:20:59 smithi067 ceph-mon[29731]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T17:21:00.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:59 smithi082 ceph-mon[33535]: pgmap v305: 97 pgs: 97 active+clean; 581 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-05-03T17:21:00.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:59 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2490503121' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-03T17:21:00.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:59 smithi082 ceph-mon[33535]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-05-03T17:21:00.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:20:59 smithi082 ceph-mon[33535]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T17:21:00.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:59 smithi070 ceph-mon[33250]: pgmap v305: 97 pgs: 97 active+clean; 581 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-05-03T17:21:00.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:59 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2490503121' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-03T17:21:00.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:59 smithi070 ceph-mon[33250]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-05-03T17:21:00.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:20:59 smithi070 ceph-mon[33250]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T17:21:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.473+0000 7fb8b972d700 1 -- 172.21.15.67:0/2908755505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b4103750 msgr2=0x7fb8b4103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.473+0000 7fb8b972d700 1 --2- 172.21.15.67:0/2908755505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b4103750 0x7fb8b4103b30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00b0d0 tx=0x7fb8ac00b3e0 comp rx=0 tx=0).stop 2024-05-03T17:21:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.473+0000 7fb8b972d700 1 -- 172.21.15.67:0/2908755505 shutdown_connections 2024-05-03T17:21:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.473+0000 7fb8b972d700 1 --2- 172.21.15.67:0/2908755505 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b4108850 0x7fb8b41103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.473+0000 7fb8b972d700 1 --2- 172.21.15.67:0/2908755505 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b4103750 0x7fb8b4103b30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.473+0000 7fb8b972d700 1 --2- 172.21.15.67:0/2908755505 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8b4104100 0x7fb8b4108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.473+0000 7fb8b972d700 1 -- 172.21.15.67:0/2908755505 >> 172.21.15.67:0/2908755505 conn(0x7fb8b407a600 msgr2=0x7fb8b407aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.473+0000 7fb8b972d700 1 -- 172.21.15.67:0/2908755505 shutdown_connections 2024-05-03T17:21:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.473+0000 7fb8b972d700 1 -- 172.21.15.67:0/2908755505 wait complete. 2024-05-03T17:21:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.473+0000 7fb8b972d700 1 Processor -- start 2024-05-03T17:21:01.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b972d700 1 -- start start 2024-05-03T17:21:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b972d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8b4103750 0x7fb8b40757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b972d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b4104100 0x7fb8b4075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b972d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b4108850 0x7fb8b4076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b972d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b4113620 con 0x7fb8b4108850 2024-05-03T17:21:01.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b972d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b41134a0 con 0x7fb8b4104100 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b972d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb8b41137a0 con 0x7fb8b4103750 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b4108850 0x7fb8b4076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b4108850 0x7fb8b4076220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33014/0 (socket says 172.21.15.67:33014) 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b37fe700 1 -- 172.21.15.67:0/2690505667 learned_addr learned my addr 172.21.15.67:0/2690505667 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b2ffd700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8b4103750 0x7fb8b40757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b27fc700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b4104100 0x7fb8b4075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b37fe700 1 -- 172.21.15.67:0/2690505667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8b4103750 msgr2=0x7fb8b40757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b37fe700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8b4103750 0x7fb8b40757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b37fe700 1 -- 172.21.15.67:0/2690505667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b4104100 msgr2=0x7fb8b4075ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b37fe700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b4104100 0x7fb8b4075ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b37fe700 1 -- 172.21.15.67:0/2690505667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8b41a8080 con 0x7fb8b4108850 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.474+0000 7fb8b37fe700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b4108850 0x7fb8b4076220 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400c970 tx=0x7fb8a400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:01.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.475+0000 7fb8abfff700 1 -- 172.21.15.67:0/2690505667 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a4004a30 con 0x7fb8b4108850 2024-05-03T17:21:01.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.475+0000 7fb8abfff700 1 -- 172.21.15.67:0/2690505667 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8a40054d0 con 0x7fb8b4108850 2024-05-03T17:21:01.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.475+0000 7fb8b972d700 1 -- 172.21.15.67:0/2690505667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8b41a8370 con 0x7fb8b4108850 2024-05-03T17:21:01.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.475+0000 7fb8abfff700 1 -- 172.21.15.67:0/2690505667 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8a400ec20 con 0x7fb8b4108850 2024-05-03T17:21:01.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.475+0000 7fb8b972d700 1 -- 172.21.15.67:0/2690505667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8b41a88b0 con 0x7fb8b4108850 2024-05-03T17:21:01.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.476+0000 7fb8b972d700 1 -- 172.21.15.67:0/2690505667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8b410ba90 con 0x7fb8b4108850 2024-05-03T17:21:01.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.477+0000 7fb8abfff700 1 -- 172.21.15.67:0/2690505667 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7fb8a4004fe0 con 0x7fb8b4108850 2024-05-03T17:21:01.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.478+0000 7fb8abfff700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb8a00b2900 0x7fb8a00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:01.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.478+0000 7fb8b2ffd700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb8a00b2900 0x7fb8a00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:01.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.478+0000 7fb8abfff700 1 -- 172.21.15.67:0/2690505667 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7fb8a40e1e60 con 0x7fb8b4108850 2024-05-03T17:21:01.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.478+0000 7fb8b2ffd700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb8a00b2900 0x7fb8a00b4dc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009fd0 tx=0x7fb8ac00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:01.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.481+0000 7fb8abfff700 1 -- 172.21.15.67:0/2690505667 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb8a4014030 con 0x7fb8b4108850 2024-05-03T17:21:01.752 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:01.752+0000 7fb8b972d700 1 -- 172.21.15.67:0/2690505667 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fb8b410b470 con 0x7fb8b4108850 2024-05-03T17:21:01.753 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:01 smithi067 ceph-mon[29731]: pgmap v306: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-05-03T17:21:02.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:01 smithi082 ceph-mon[33535]: pgmap v306: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-05-03T17:21:02.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:01 smithi070 ceph-mon[33250]: pgmap v306: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-05-03T17:21:02.573 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.572+0000 7fb8abfff700 1 -- 172.21.15.67:0/2690505667 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v25) ==== 115+0+0 (secure 0 0 0) 0x7fb8a40a9d50 con 0x7fb8b4108850 2024-05-03T17:21:02.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.575+0000 7fb8b972d700 1 -- 172.21.15.67:0/2690505667 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb8a00b2900 msgr2=0x7fb8a00b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:02.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.575+0000 7fb8b972d700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb8a00b2900 0x7fb8a00b4dc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009fd0 tx=0x7fb8ac00b040 comp rx=0 tx=0).stop 2024-05-03T17:21:02.576 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.575+0000 7fb8b972d700 1 -- 172.21.15.67:0/2690505667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b4108850 msgr2=0x7fb8b4076220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.575+0000 7fb8b972d700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b4108850 0x7fb8b4076220 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb8a400c970 tx=0x7fb8a400cc80 comp rx=0 tx=0).stop 2024-05-03T17:21:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.576+0000 7fb8b972d700 1 -- 172.21.15.67:0/2690505667 shutdown_connections 2024-05-03T17:21:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.576+0000 7fb8b972d700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb8a00b2900 0x7fb8a00b4dc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.576+0000 7fb8b972d700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb8b4108850 0x7fb8b4076220 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.576+0000 7fb8b972d700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb8b4103750 0x7fb8b40757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.576+0000 7fb8b972d700 1 --2- 172.21.15.67:0/2690505667 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb8b4104100 0x7fb8b4075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.576+0000 7fb8b972d700 1 -- 172.21.15.67:0/2690505667 >> 172.21.15.67:0/2690505667 conn(0x7fb8b407a600 msgr2=0x7fb8b41099a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.576+0000 7fb8b972d700 1 -- 172.21.15.67:0/2690505667 shutdown_connections 2024-05-03T17:21:02.577 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:02.576+0000 7fb8b972d700 1 -- 172.21.15.67:0/2690505667 wait complete. 2024-05-03T17:21:02.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:02 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2690505667' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-03T17:21:02.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:02 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2690505667' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-03T17:21:03.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:02 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2690505667' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-03T17:21:03.427 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-05-03T17:21:03.427 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-05-03T17:21:03.677 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:03.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:03 smithi067 ceph-mon[29731]: pgmap v307: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-05-03T17:21:03.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:03 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2690505667' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-03T17:21:03.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:03 smithi067 ceph-mon[29731]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T17:21:04.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:03 smithi082 ceph-mon[33535]: pgmap v307: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-05-03T17:21:04.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:03 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2690505667' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-03T17:21:04.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:03 smithi082 ceph-mon[33535]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T17:21:04.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:03 smithi070 ceph-mon[33250]: pgmap v307: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 2.0 KiB/s wr, 7 op/s 2024-05-03T17:21:04.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:03 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2690505667' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-03T17:21:04.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:03 smithi070 ceph-mon[33250]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T17:21:05.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.425+0000 7ff2531ae700 1 -- 172.21.15.67:0/976078353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c104130 msgr2=0x7ff24c108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:05.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.425+0000 7ff2531ae700 1 --2- 172.21.15.67:0/976078353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c104130 0x7ff24c108140 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff23c0099c0 tx=0x7ff23c009cd0 comp rx=0 tx=0).stop 2024-05-03T17:21:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.425+0000 7ff2531ae700 1 -- 172.21.15.67:0/976078353 shutdown_connections 2024-05-03T17:21:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.425+0000 7ff2531ae700 1 --2- 172.21.15.67:0/976078353 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff24c108870 0x7ff24c1103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.425+0000 7ff2531ae700 1 --2- 172.21.15.67:0/976078353 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff24c103780 0x7ff24c103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.425+0000 7ff2531ae700 1 --2- 172.21.15.67:0/976078353 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c104130 0x7ff24c108140 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.425+0000 7ff2531ae700 1 -- 172.21.15.67:0/976078353 >> 172.21.15.67:0/976078353 conn(0x7ff24c07a610 msgr2=0x7ff24c07aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.426+0000 7ff2531ae700 1 -- 172.21.15.67:0/976078353 shutdown_connections 2024-05-03T17:21:05.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.426+0000 7ff2531ae700 1 -- 172.21.15.67:0/976078353 wait complete. 2024-05-03T17:21:05.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.426+0000 7ff2531ae700 1 Processor -- start 2024-05-03T17:21:05.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.426+0000 7ff2531ae700 1 -- start start 2024-05-03T17:21:05.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff2531ae700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff24c103780 0x7ff24c0757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:05.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff2531ae700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c104130 0x7ff24c075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:05.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff2531ae700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff24c108870 0x7ff24c076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:05.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff2531ae700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c113680 con 0x7ff24c104130 2024-05-03T17:21:05.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff2531ae700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c113500 con 0x7ff24c103780 2024-05-03T17:21:05.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff2531ae700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff24c113800 con 0x7ff24c108870 2024-05-03T17:21:05.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff24bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c104130 0x7ff24c075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:05.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff24bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c104130 0x7ff24c075ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:33032/0 (socket says 172.21.15.67:33032) 2024-05-03T17:21:05.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff24bfff700 1 -- 172.21.15.67:0/202945445 learned_addr learned my addr 172.21.15.67:0/202945445 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:05.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff250f4a700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff24c103780 0x7ff24c0757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:05.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff25174b700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff24c108870 0x7ff24c076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:05.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff24bfff700 1 -- 172.21.15.67:0/202945445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff24c108870 msgr2=0x7ff24c076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:05.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff24bfff700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff24c108870 0x7ff24c076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:05.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff24bfff700 1 -- 172.21.15.67:0/202945445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff24c103780 msgr2=0x7ff24c0757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:05.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff24bfff700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff24c103780 0x7ff24c0757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:05.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff24bfff700 1 -- 172.21.15.67:0/202945445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24c1a7f90 con 0x7ff24c104130 2024-05-03T17:21:05.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.427+0000 7ff24bfff700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c104130 0x7ff24c075ce0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009990 tx=0x7ff23c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:05.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.428+0000 7ff249ffb700 1 -- 172.21.15.67:0/202945445 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c027070 con 0x7ff24c104130 2024-05-03T17:21:05.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.428+0000 7ff2531ae700 1 -- 172.21.15.67:0/202945445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24c1a8220 con 0x7ff24c104130 2024-05-03T17:21:05.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.428+0000 7ff249ffb700 1 -- 172.21.15.67:0/202945445 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff23c0040a0 con 0x7ff24c104130 2024-05-03T17:21:05.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.428+0000 7ff249ffb700 1 -- 172.21.15.67:0/202945445 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23c022e60 con 0x7ff24c104130 2024-05-03T17:21:05.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.428+0000 7ff2531ae700 1 -- 172.21.15.67:0/202945445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff24c1a86d0 con 0x7ff24c104130 2024-05-03T17:21:05.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.429+0000 7ff2531ae700 1 -- 172.21.15.67:0/202945445 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff24c06a190 con 0x7ff24c104130 2024-05-03T17:21:05.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.430+0000 7ff249ffb700 1 -- 172.21.15.67:0/202945445 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 16) ==== 149652+0+0 (secure 0 0 0) 0x7ff23c02a070 con 0x7ff24c104130 2024-05-03T17:21:05.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.431+0000 7ff249ffb700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff2380b2880 0x7ff2380b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:05.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.431+0000 7ff250f4a700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff2380b2880 0x7ff2380b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:05.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.431+0000 7ff249ffb700 1 -- 172.21.15.67:0/202945445 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7ff23c0e2e50 con 0x7ff24c104130 2024-05-03T17:21:05.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.431+0000 7ff250f4a700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff2380b2880 0x7ff2380b4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff23400ac50 tx=0x7ff23400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:05.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.436+0000 7ff249ffb700 1 -- 172.21.15.67:0/202945445 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ff23c013070 con 0x7ff24c104130 2024-05-03T17:21:05.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.681+0000 7ff2531ae700 1 -- 172.21.15.67:0/202945445 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ff24c0638e0 con 0x7ff2380b2880 2024-05-03T17:21:05.683 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:05 smithi067 ceph-mon[29731]: pgmap v308: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-03T17:21:05.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.705+0000 7ff249ffb700 1 -- 172.21.15.67:0/202945445 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ff24c0638e0 con 0x7ff2380b2880 2024-05-03T17:21:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.707+0000 7ff2531ae700 1 -- 172.21.15.67:0/202945445 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff2380b2880 msgr2=0x7ff2380b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.707+0000 7ff2531ae700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff2380b2880 0x7ff2380b4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff23400ac50 tx=0x7ff23400a380 comp rx=0 tx=0).stop 2024-05-03T17:21:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.707+0000 7ff2531ae700 1 -- 172.21.15.67:0/202945445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c104130 msgr2=0x7ff24c075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.707+0000 7ff2531ae700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c104130 0x7ff24c075ce0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff23c009990 tx=0x7ff23c0047b0 comp rx=0 tx=0).stop 2024-05-03T17:21:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.707+0000 7ff2531ae700 1 -- 172.21.15.67:0/202945445 shutdown_connections 2024-05-03T17:21:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.707+0000 7ff2531ae700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff2380b2880 0x7ff2380b4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.707+0000 7ff2531ae700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff24c108870 0x7ff24c076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.707+0000 7ff2531ae700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff24c103780 0x7ff24c0757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.707+0000 7ff2531ae700 1 --2- 172.21.15.67:0/202945445 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff24c104130 0x7ff24c075ce0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:05.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.707+0000 7ff2531ae700 1 -- 172.21.15.67:0/202945445 >> 172.21.15.67:0/202945445 conn(0x7ff24c07a610 msgr2=0x7ff24c1014d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:05.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.707+0000 7ff2531ae700 1 -- 172.21.15.67:0/202945445 shutdown_connections 2024-05-03T17:21:05.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:05.707+0000 7ff2531ae700 1 -- 172.21.15.67:0/202945445 wait complete. 2024-05-03T17:21:05.953 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:21:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:21:05.686+0000 7f831f900700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-03T17:21:05.953 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:21:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:21:05.686+0000 7f831f900700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-03T17:21:05.953 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:21:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:21:05.686+0000 7f831f900700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-03T17:21:05.953 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:21:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:21:05.686+0000 7f831f900700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-03T17:21:05.954 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:21:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:21:05.686+0000 7f831f900700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-03T17:21:06.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:05 smithi082 ceph-mon[33535]: pgmap v308: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-03T17:21:06.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:05 smithi070 ceph-mon[33250]: pgmap v308: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-03T17:21:06.350 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-03T17:21:06.618 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:06.883 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:06 smithi067 ceph-mon[29731]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:06.883 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-05-03T17:21:06.883 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11, 2, 1]}]: dispatch 2024-05-03T17:21:06.883 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-05-03T17:21:06.883 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-05-03T17:21:06.883 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-05-03T17:21:06.883 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-05-03T17:21:06.883 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-05-03T17:21:06.883 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-05-03T17:21:06.883 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:06 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-05-03T17:21:07.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:06 smithi082 ceph-mon[33535]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:07.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-05-03T17:21:07.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11, 2, 1]}]: dispatch 2024-05-03T17:21:07.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-05-03T17:21:07.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-05-03T17:21:07.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-05-03T17:21:07.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-05-03T17:21:07.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-05-03T17:21:07.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-05-03T17:21:07.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:06 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-05-03T17:21:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:06 smithi070 ceph-mon[33250]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-05-03T17:21:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11, 2, 1]}]: dispatch 2024-05-03T17:21:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-05-03T17:21:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]: dispatch 2024-05-03T17:21:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]: dispatch 2024-05-03T17:21:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]: dispatch 2024-05-03T17:21:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]: dispatch 2024-05-03T17:21:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]: dispatch 2024-05-03T17:21:07.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:06 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]: dispatch 2024-05-03T17:21:07.772 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:07 smithi067 ceph-mon[29731]: pgmap v309: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-03T17:21:07.772 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-05-03T17:21:07.772 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11, 2, 1]}]': finished 2024-05-03T17:21:07.772 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-05-03T17:21:07.772 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-05-03T17:21:07.772 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-05-03T17:21:07.772 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-05-03T17:21:07.772 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-05-03T17:21:07.772 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-05-03T17:21:07.773 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:07 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-05-03T17:21:07.773 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:07 smithi067 ceph-mon[29731]: osdmap e80: 12 total, 12 up, 12 in 2024-05-03T17:21:07.773 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:07 smithi067 ceph-mon[29731]: mgrmap e17: x(active, since 8m), standbys: y, z 2024-05-03T17:21:08.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:07 smithi082 ceph-mon[33535]: pgmap v309: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-03T17:21:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-05-03T17:21:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11, 2, 1]}]': finished 2024-05-03T17:21:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-05-03T17:21:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-05-03T17:21:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-05-03T17:21:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-05-03T17:21:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-05-03T17:21:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-05-03T17:21:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:07 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-05-03T17:21:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:07 smithi082 ceph-mon[33535]: osdmap e80: 12 total, 12 up, 12 in 2024-05-03T17:21:08.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:07 smithi082 ceph-mon[33535]: mgrmap e17: x(active, since 8m), standbys: y, z 2024-05-03T17:21:08.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:07 smithi070 ceph-mon[33250]: pgmap v309: 97 pgs: 97 active+clean; 582 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 2.0 KiB/s wr, 10 op/s 2024-05-03T17:21:08.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-05-03T17:21:08.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [8, 11, 2, 1]}]': finished 2024-05-03T17:21:08.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-05-03T17:21:08.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 8]}]': finished 2024-05-03T17:21:08.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.19", "id": [3, 7]}]': finished 2024-05-03T17:21:08.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [5, 9]}]': finished 2024-05-03T17:21:08.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.24", "id": [4, 9]}]': finished 2024-05-03T17:21:08.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.33", "id": [5, 9]}]': finished 2024-05-03T17:21:08.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:07 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 6]}]': finished 2024-05-03T17:21:08.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:07 smithi070 ceph-mon[33250]: osdmap e80: 12 total, 12 up, 12 in 2024-05-03T17:21:08.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:07 smithi070 ceph-mon[33250]: mgrmap e17: x(active, since 8m), standbys: y, z 2024-05-03T17:21:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.344+0000 7feeb9e0f700 1 -- 172.21.15.67:0/2176226478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb406bc70 msgr2=0x7feeb406c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.344+0000 7feeb9e0f700 1 --2- 172.21.15.67:0/2176226478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb406bc70 0x7feeb406c0f0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7feea80099c0 tx=0x7feea8009cd0 comp rx=0 tx=0).stop 2024-05-03T17:21:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.344+0000 7feeb9e0f700 1 -- 172.21.15.67:0/2176226478 shutdown_connections 2024-05-03T17:21:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.344+0000 7feeb9e0f700 1 --2- 172.21.15.67:0/2176226478 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb406c630 0x7feeb4110360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.344+0000 7feeb9e0f700 1 --2- 172.21.15.67:0/2176226478 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feeb4109590 0x7feeb4109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.344+0000 7feeb9e0f700 1 --2- 172.21.15.67:0/2176226478 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb406bc70 0x7feeb406c0f0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.344+0000 7feeb9e0f700 1 -- 172.21.15.67:0/2176226478 >> 172.21.15.67:0/2176226478 conn(0x7feeb4079440 msgr2=0x7feeb4079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:08.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.344+0000 7feeb9e0f700 1 -- 172.21.15.67:0/2176226478 shutdown_connections 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.344+0000 7feeb9e0f700 1 -- 172.21.15.67:0/2176226478 wait complete. 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.344+0000 7feeb9e0f700 1 Processor -- start 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb9e0f700 1 -- start start 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb9e0f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb406bc70 0x7feeb4102410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb9e0f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feeb406c630 0x7feeb4102950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb9e0f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb4109590 0x7feeb4102e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb9e0f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feeb41136f0 con 0x7feeb406bc70 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb9e0f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feeb4113570 con 0x7feeb4109590 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb9e0f700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7feeb4113870 con 0x7feeb406c630 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb406bc70 0x7feeb4102410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb37fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb406bc70 0x7feeb4102410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47922/0 (socket says 172.21.15.67:47922) 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb37fe700 1 -- 172.21.15.67:0/3365840769 learned_addr learned my addr 172.21.15.67:0/3365840769 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb2ffd700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feeb406c630 0x7feeb4102950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb3fff700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb4109590 0x7feeb4102e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb37fe700 1 -- 172.21.15.67:0/3365840769 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feeb406c630 msgr2=0x7feeb4102950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:08.348 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb37fe700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feeb406c630 0x7feeb4102950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:08.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb37fe700 1 -- 172.21.15.67:0/3365840769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb4109590 msgr2=0x7feeb4102e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:08.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb37fe700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb4109590 0x7feeb4102e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:08.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb37fe700 1 -- 172.21.15.67:0/3365840769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeb41a8080 con 0x7feeb406bc70 2024-05-03T17:21:08.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.345+0000 7feeb3fff700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb4109590 0x7feeb4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:21:08.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.346+0000 7feeb37fe700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb406bc70 0x7feeb4102410 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feea400c920 tx=0x7feea400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:08.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.346+0000 7feeb0ff9700 1 -- 172.21.15.67:0/3365840769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4007b00 con 0x7feeb406bc70 2024-05-03T17:21:08.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.346+0000 7feeb0ff9700 1 -- 172.21.15.67:0/3365840769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feea4004ce0 con 0x7feeb406bc70 2024-05-03T17:21:08.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.346+0000 7feeb9e0f700 1 -- 172.21.15.67:0/3365840769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeb41a8370 con 0x7feeb406bc70 2024-05-03T17:21:08.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.346+0000 7feeb0ff9700 1 -- 172.21.15.67:0/3365840769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4005550 con 0x7feeb406bc70 2024-05-03T17:21:08.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.346+0000 7feeb9e0f700 1 -- 172.21.15.67:0/3365840769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feeb41a88d0 con 0x7feeb406bc70 2024-05-03T17:21:08.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.348+0000 7feeb9e0f700 1 -- 172.21.15.67:0/3365840769 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feeb406a190 con 0x7feeb406bc70 2024-05-03T17:21:08.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.349+0000 7feeb0ff9700 1 -- 172.21.15.67:0/3365840769 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7feea4007ca0 con 0x7feeb406bc70 2024-05-03T17:21:08.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.349+0000 7feeb0ff9700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fee9c0b2990 0x7fee9c0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:08.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.350+0000 7feeb2ffd700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fee9c0b2990 0x7fee9c0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:08.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.350+0000 7feeb0ff9700 1 -- 172.21.15.67:0/3365840769 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7feea40e2220 con 0x7feeb406bc70 2024-05-03T17:21:08.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.350+0000 7feeb2ffd700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fee9c0b2990 0x7fee9c0b4e50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feea8009990 tx=0x7feea8009650 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:08.354 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.354+0000 7feeb0ff9700 1 -- 172.21.15.67:0/3365840769 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7feea4014030 con 0x7feeb406bc70 2024-05-03T17:21:08.603 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.602+0000 7feeb9e0f700 1 -- 172.21.15.67:0/3365840769 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7feeb40638e0 con 0x7fee9c0b2990 2024-05-03T17:21:08.604 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:08 smithi067 ceph-mon[29731]: osdmap e81: 12 total, 12 up, 12 in 2024-05-03T17:21:08.630 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.630+0000 7feeb0ff9700 1 -- 172.21.15.67:0/3365840769 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7feeb40638e0 con 0x7fee9c0b2990 2024-05-03T17:21:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.632+0000 7feeb9e0f700 1 -- 172.21.15.67:0/3365840769 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fee9c0b2990 msgr2=0x7fee9c0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.632+0000 7feeb9e0f700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fee9c0b2990 0x7fee9c0b4e50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feea8009990 tx=0x7feea8009650 comp rx=0 tx=0).stop 2024-05-03T17:21:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.632+0000 7feeb9e0f700 1 -- 172.21.15.67:0/3365840769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb406bc70 msgr2=0x7feeb4102410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.632+0000 7feeb9e0f700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb406bc70 0x7feeb4102410 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feea400c920 tx=0x7feea400cc30 comp rx=0 tx=0).stop 2024-05-03T17:21:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.632+0000 7feeb9e0f700 1 -- 172.21.15.67:0/3365840769 shutdown_connections 2024-05-03T17:21:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.632+0000 7feeb9e0f700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fee9c0b2990 0x7fee9c0b4e50 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.632+0000 7feeb9e0f700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feeb4109590 0x7feeb4102e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.632+0000 7feeb9e0f700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feeb406bc70 0x7feeb4102410 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.632+0000 7feeb9e0f700 1 --2- 172.21.15.67:0/3365840769 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feeb406c630 0x7feeb4102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.632+0000 7feeb9e0f700 1 -- 172.21.15.67:0/3365840769 >> 172.21.15.67:0/3365840769 conn(0x7feeb4079440 msgr2=0x7feeb41010c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.632+0000 7feeb9e0f700 1 -- 172.21.15.67:0/3365840769 shutdown_connections 2024-05-03T17:21:08.633 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:08.632+0000 7feeb9e0f700 1 -- 172.21.15.67:0/3365840769 wait complete. 2024-05-03T17:21:09.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:08 smithi082 ceph-mon[33535]: osdmap e81: 12 total, 12 up, 12 in 2024-05-03T17:21:09.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:08 smithi070 ceph-mon[33250]: osdmap e81: 12 total, 12 up, 12 in 2024-05-03T17:21:09.353 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-03T17:21:09.604 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:09.870 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:09 smithi067 ceph-mon[29731]: pgmap v312: 97 pgs: 1 peering, 96 active+clean; 588 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-05-03T17:21:09.871 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:09 smithi067 ceph-mon[29731]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:10.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:09 smithi082 ceph-mon[33535]: pgmap v312: 97 pgs: 1 peering, 96 active+clean; 588 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-05-03T17:21:10.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:09 smithi082 ceph-mon[33535]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:10.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:09 smithi070 ceph-mon[33250]: pgmap v312: 97 pgs: 1 peering, 96 active+clean; 588 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 9 op/s 2024-05-03T17:21:10.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:09 smithi070 ceph-mon[33250]: from='client.14667 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.345+0000 7f996c2c6700 1 -- 172.21.15.67:0/1032305345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9964102fe0 msgr2=0x7f9964106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:11.346 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.345+0000 7f996c2c6700 1 --2- 172.21.15.67:0/1032305345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9964102fe0 0x7f9964106fe0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f99540099c0 tx=0x7f9954009cd0 comp rx=0 tx=0).stop 2024-05-03T17:21:11.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.345+0000 7f996c2c6700 1 -- 172.21.15.67:0/1032305345 shutdown_connections 2024-05-03T17:21:11.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.345+0000 7f996c2c6700 1 --2- 172.21.15.67:0/1032305345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f99641076c0 0x7f996410f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:11.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.345+0000 7f996c2c6700 1 --2- 172.21.15.67:0/1032305345 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9964102630 0x7f9964102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:11.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.345+0000 7f996c2c6700 1 --2- 172.21.15.67:0/1032305345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9964102fe0 0x7f9964106fe0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:11.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.345+0000 7f996c2c6700 1 -- 172.21.15.67:0/1032305345 >> 172.21.15.67:0/1032305345 conn(0x7f9964079350 msgr2=0x7f9964079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:11.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.346+0000 7f996c2c6700 1 -- 172.21.15.67:0/1032305345 shutdown_connections 2024-05-03T17:21:11.347 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.346+0000 7f996c2c6700 1 -- 172.21.15.67:0/1032305345 wait complete. 2024-05-03T17:21:11.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.346+0000 7f996c2c6700 1 Processor -- start 2024-05-03T17:21:11.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.346+0000 7f996c2c6700 1 -- start start 2024-05-03T17:21:11.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996c2c6700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9964102630 0x7f996419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:11.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996c2c6700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9964102fe0 0x7f996419ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:11.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996c2c6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99641076c0 0x7f9964198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:11.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996c2c6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f99641125b0 con 0x7f99641076c0 2024-05-03T17:21:11.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996c2c6700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9964112430 con 0x7f9964102fe0 2024-05-03T17:21:11.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996c2c6700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9964112730 con 0x7f9964102630 2024-05-03T17:21:11.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996a863700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99641076c0 0x7f9964198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:11.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996a062700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9964102630 0x7f996419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:11.349 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996a863700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99641076c0 0x7f9964198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47938/0 (socket says 172.21.15.67:47938) 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996a062700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9964102630 0x7f996419e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:53110/0 (socket says 172.21.15.67:53110) 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996a863700 1 -- 172.21.15.67:0/4272441797 learned_addr learned my addr 172.21.15.67:0/4272441797 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f9969861700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9964102fe0 0x7f996419ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996a863700 1 -- 172.21.15.67:0/4272441797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9964102630 msgr2=0x7f996419e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996a863700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9964102630 0x7f996419e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996a863700 1 -- 172.21.15.67:0/4272441797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9964102fe0 msgr2=0x7f996419ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996a863700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9964102fe0 0x7f996419ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996a863700 1 -- 172.21.15.67:0/4272441797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9964199090 con 0x7f99641076c0 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f9969861700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9964102fe0 0x7f996419ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.347+0000 7f996a863700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99641076c0 0x7f9964198930 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f995c00b970 tx=0x7f995c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.348+0000 7f995b7fe700 1 -- 172.21.15.67:0/4272441797 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c019070 con 0x7f99641076c0 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.348+0000 7f996c2c6700 1 -- 172.21.15.67:0/4272441797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9964199380 con 0x7f99641076c0 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.348+0000 7f995b7fe700 1 -- 172.21.15.67:0/4272441797 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f995c00cde0 con 0x7f99641076c0 2024-05-03T17:21:11.350 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.348+0000 7f995b7fe700 1 -- 172.21.15.67:0/4272441797 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f995c007370 con 0x7f99641076c0 2024-05-03T17:21:11.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.348+0000 7f996c2c6700 1 -- 172.21.15.67:0/4272441797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f996406bbd0 con 0x7f99641076c0 2024-05-03T17:21:11.351 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.349+0000 7f996c2c6700 1 -- 172.21.15.67:0/4272441797 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9964103ac0 con 0x7f99641076c0 2024-05-03T17:21:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.350+0000 7f995b7fe700 1 -- 172.21.15.67:0/4272441797 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f995c01c070 con 0x7f99641076c0 2024-05-03T17:21:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.351+0000 7f995b7fe700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f99500b4ba0 0x7f99500b7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.351+0000 7f996a062700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f99500b4ba0 0x7f99500b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.351+0000 7f995b7fe700 1 -- 172.21.15.67:0/4272441797 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f995c0e1750 con 0x7f99641076c0 2024-05-03T17:21:11.352 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.351+0000 7f996a062700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f99500b4ba0 0x7f99500b7060 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9960005950 tx=0x7f996000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:11.356 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.356+0000 7f995b7fe700 1 -- 172.21.15.67:0/4272441797 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f995c051d70 con 0x7f99641076c0 2024-05-03T17:21:11.605 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.605+0000 7f996c2c6700 1 -- 172.21.15.67:0/4272441797 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f996406bef0 con 0x7f99500b4ba0 2024-05-03T17:21:11.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.610+0000 7f995b7fe700 1 -- 172.21.15.67:0/4272441797 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f996406bef0 con 0x7f99500b4ba0 2024-05-03T17:21:11.611 INFO:teuthology.orchestra.run.smithi067.stdout:/volumes/qa/sv_0/e56b02b3-e8d9-43c3-84f8-ffcd7359bb38 2024-05-03T17:21:11.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.612+0000 7f996c2c6700 1 -- 172.21.15.67:0/4272441797 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f99500b4ba0 msgr2=0x7f99500b7060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:11.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.612+0000 7f996c2c6700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f99500b4ba0 0x7f99500b7060 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9960005950 tx=0x7f996000a300 comp rx=0 tx=0).stop 2024-05-03T17:21:11.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.612+0000 7f996c2c6700 1 -- 172.21.15.67:0/4272441797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99641076c0 msgr2=0x7f9964198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:11.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.612+0000 7f996c2c6700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99641076c0 0x7f9964198930 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f995c00b970 tx=0x7f995c00bc80 comp rx=0 tx=0).stop 2024-05-03T17:21:11.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.612+0000 7f996c2c6700 1 -- 172.21.15.67:0/4272441797 shutdown_connections 2024-05-03T17:21:11.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.612+0000 7f996c2c6700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f99500b4ba0 0x7f99500b7060 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:11.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.612+0000 7f996c2c6700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f99641076c0 0x7f9964198930 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:11.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.612+0000 7f996c2c6700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9964102630 0x7f996419e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:11.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.612+0000 7f996c2c6700 1 --2- 172.21.15.67:0/4272441797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9964102fe0 0x7f996419ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:11.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.612+0000 7f996c2c6700 1 -- 172.21.15.67:0/4272441797 >> 172.21.15.67:0/4272441797 conn(0x7f9964079350 msgr2=0x7f9964100630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:11.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.613+0000 7f996c2c6700 1 -- 172.21.15.67:0/4272441797 shutdown_connections 2024-05-03T17:21:11.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:11.613+0000 7f996c2c6700 1 -- 172.21.15.67:0/4272441797 wait complete. 2024-05-03T17:21:11.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:11 smithi067 ceph-mon[29731]: pgmap v313: 97 pgs: 1 peering, 96 active+clean; 588 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.4 KiB/s wr, 8 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:12.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:11 smithi082 ceph-mon[33535]: pgmap v313: 97 pgs: 1 peering, 96 active+clean; 588 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.4 KiB/s wr, 8 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:12.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:11 smithi070 ceph-mon[33250]: pgmap v313: 97 pgs: 1 peering, 96 active+clean; 588 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.4 KiB/s wr, 8 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:12.299 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-03T17:21:12.551 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:12.935 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:12 smithi067 ceph-mon[29731]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:13.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:12 smithi082 ceph-mon[33535]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:13.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:12 smithi070 ceph-mon[33250]: from='client.14673 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:13.875 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:13 smithi067 ceph-mon[29731]: pgmap v314: 97 pgs: 1 peering, 96 active+clean; 588 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.4 KiB/s wr, 5 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:14.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:13 smithi082 ceph-mon[33535]: pgmap v314: 97 pgs: 1 peering, 96 active+clean; 588 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.4 KiB/s wr, 5 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:13 smithi070 ceph-mon[33250]: pgmap v314: 97 pgs: 1 peering, 96 active+clean; 588 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.4 KiB/s wr, 5 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.463+0000 7f42bb38b700 1 -- 172.21.15.67:0/384307739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b4109520 msgr2=0x7f42b4109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.463+0000 7f42bb38b700 1 --2- 172.21.15.67:0/384307739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b4109520 0x7f42b4109900 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00b0a0 tx=0x7f42ac00b3b0 comp rx=0 tx=0).stop 2024-05-03T17:21:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.463+0000 7f42bb38b700 1 -- 172.21.15.67:0/384307739 shutdown_connections 2024-05-03T17:21:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.463+0000 7f42bb38b700 1 --2- 172.21.15.67:0/384307739 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42b4102450 0x7f42b4110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.463+0000 7f42bb38b700 1 --2- 172.21.15.67:0/384307739 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b4109520 0x7f42b4109900 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.463+0000 7f42bb38b700 1 --2- 172.21.15.67:0/384307739 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42b4101a90 0x7f42b4101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:14.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.463+0000 7f42bb38b700 1 -- 172.21.15.67:0/384307739 >> 172.21.15.67:0/384307739 conn(0x7f42b4078a10 msgr2=0x7f42b40ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.463+0000 7f42bb38b700 1 -- 172.21.15.67:0/384307739 shutdown_connections 2024-05-03T17:21:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.463+0000 7f42bb38b700 1 -- 172.21.15.67:0/384307739 wait complete. 2024-05-03T17:21:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42bb38b700 1 Processor -- start 2024-05-03T17:21:14.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42bb38b700 1 -- start start 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42bb38b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42b4101a90 0x7f42b4102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42bb38b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b4102450 0x7f42b41031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42bb38b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42b4109520 0x7f42b41036f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42bb38b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f42b4113630 con 0x7f42b4102450 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42bb38b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f42b41134b0 con 0x7f42b4101a90 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42bb38b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f42b41137b0 con 0x7f42b4109520 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42b8926700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b4102450 0x7f42b41031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42b8926700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b4102450 0x7f42b41031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:47952/0 (socket says 172.21.15.67:47952) 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42b8926700 1 -- 172.21.15.67:0/3476917206 learned_addr learned my addr 172.21.15.67:0/3476917206 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42b9127700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42b4101a90 0x7f42b4102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42b9928700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42b4109520 0x7f42b41036f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42b8926700 1 -- 172.21.15.67:0/3476917206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42b4109520 msgr2=0x7f42b41036f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42b8926700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42b4109520 0x7f42b41036f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:14.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42b8926700 1 -- 172.21.15.67:0/3476917206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42b4101a90 msgr2=0x7f42b4102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42b8926700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42b4101a90 0x7f42b4102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42b8926700 1 -- 172.21.15.67:0/3476917206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42b41a8150 con 0x7f42b4102450 2024-05-03T17:21:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.464+0000 7f42b9127700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42b4101a90 0x7f42b4102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:21:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.465+0000 7f42b8926700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b4102450 0x7f42b41031b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f42a800d980 tx=0x7f42a800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.465+0000 7f42b27fc700 1 -- 172.21.15.67:0/3476917206 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a8016070 con 0x7f42b4102450 2024-05-03T17:21:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.465+0000 7f42bb38b700 1 -- 172.21.15.67:0/3476917206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42b41a8440 con 0x7f42b4102450 2024-05-03T17:21:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.465+0000 7f42b27fc700 1 -- 172.21.15.67:0/3476917206 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42a80044d0 con 0x7f42b4102450 2024-05-03T17:21:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.465+0000 7f42bb38b700 1 -- 172.21.15.67:0/3476917206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42b41a89d0 con 0x7f42b4102450 2024-05-03T17:21:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.465+0000 7f42b27fc700 1 -- 172.21.15.67:0/3476917206 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42a8004d00 con 0x7f42b4102450 2024-05-03T17:21:14.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.466+0000 7f42bb38b700 1 -- 172.21.15.67:0/3476917206 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42a0005330 con 0x7f42b4102450 2024-05-03T17:21:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.467+0000 7f42b27fc700 1 -- 172.21.15.67:0/3476917206 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f42a800bb60 con 0x7f42b4102450 2024-05-03T17:21:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.468+0000 7f42b27fc700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f429c0b2990 0x7f429c0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.468+0000 7f42b9127700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f429c0b2990 0x7f429c0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.468+0000 7f42b27fc700 1 -- 172.21.15.67:0/3476917206 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f42a80e20e0 con 0x7f42b4102450 2024-05-03T17:21:14.469 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.469+0000 7f42b9127700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f429c0b2990 0x7f429c0b4e50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009e60 tx=0x7f42ac006230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:14.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.473+0000 7f42b27fc700 1 -- 172.21.15.67:0/3476917206 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f42a801f020 con 0x7f42b4102450 2024-05-03T17:21:14.721 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.720+0000 7f42bb38b700 1 -- 172.21.15.67:0/3476917206 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f42a0000bc0 con 0x7f429c0b2990 2024-05-03T17:21:14.746 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.746+0000 7f42b27fc700 1 -- 172.21.15.67:0/3476917206 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f42a0000bc0 con 0x7f429c0b2990 2024-05-03T17:21:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.748+0000 7f42bb38b700 1 -- 172.21.15.67:0/3476917206 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f429c0b2990 msgr2=0x7f429c0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.748+0000 7f42bb38b700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f429c0b2990 0x7f429c0b4e50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f42ac009e60 tx=0x7f42ac006230 comp rx=0 tx=0).stop 2024-05-03T17:21:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.748+0000 7f42bb38b700 1 -- 172.21.15.67:0/3476917206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b4102450 msgr2=0x7f42b41031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.748+0000 7f42bb38b700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b4102450 0x7f42b41031b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f42a800d980 tx=0x7f42a800dd40 comp rx=0 tx=0).stop 2024-05-03T17:21:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.748+0000 7f42bb38b700 1 -- 172.21.15.67:0/3476917206 shutdown_connections 2024-05-03T17:21:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.748+0000 7f42bb38b700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f429c0b2990 0x7f429c0b4e50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.748+0000 7f42bb38b700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f42b4109520 0x7f42b41036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.748+0000 7f42bb38b700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f42b4101a90 0x7f42b4102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.748+0000 7f42bb38b700 1 --2- 172.21.15.67:0/3476917206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f42b4102450 0x7f42b41031b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.748+0000 7f42bb38b700 1 -- 172.21.15.67:0/3476917206 >> 172.21.15.67:0/3476917206 conn(0x7f42b4078a10 msgr2=0x7f42b41002e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.748+0000 7f42bb38b700 1 -- 172.21.15.67:0/3476917206 shutdown_connections 2024-05-03T17:21:14.749 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:14.748+0000 7f42bb38b700 1 -- 172.21.15.67:0/3476917206 wait complete. 2024-05-03T17:21:15.519 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-03T17:21:15.778 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:15.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:15 smithi067 ceph-mon[29731]: pgmap v315: 97 pgs: 97 active+clean; 602 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.4 KiB/s wr, 9 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:15.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:15 smithi067 ceph-mon[29731]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:16.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:15 smithi082 ceph-mon[33535]: pgmap v315: 97 pgs: 97 active+clean; 602 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.4 KiB/s wr, 9 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:16.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:15 smithi082 ceph-mon[33535]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:16.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:15 smithi070 ceph-mon[33250]: pgmap v315: 97 pgs: 97 active+clean; 602 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.4 KiB/s wr, 9 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:16.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:15 smithi070 ceph-mon[33250]: from='client.14679 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.472+0000 7f454d5db700 1 -- 172.21.15.67:0/400864170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548109520 msgr2=0x7f4548109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.472+0000 7f454d5db700 1 --2- 172.21.15.67:0/400864170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548109520 0x7f4548109900 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f453800b0a0 tx=0x7f453800b3b0 comp rx=0 tx=0).stop 2024-05-03T17:21:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.472+0000 7f454d5db700 1 -- 172.21.15.67:0/400864170 shutdown_connections 2024-05-03T17:21:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.472+0000 7f454d5db700 1 --2- 172.21.15.67:0/400864170 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4548102450 0x7f4548110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.472+0000 7f454d5db700 1 --2- 172.21.15.67:0/400864170 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548109520 0x7f4548109900 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:17.473 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.472+0000 7f454d5db700 1 --2- 172.21.15.67:0/400864170 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4548101a90 0x7f4548101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.472+0000 7f454d5db700 1 -- 172.21.15.67:0/400864170 >> 172.21.15.67:0/400864170 conn(0x7f4548078a10 msgr2=0x7f45480ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.472+0000 7f454d5db700 1 -- 172.21.15.67:0/400864170 shutdown_connections 2024-05-03T17:21:17.474 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.472+0000 7f454d5db700 1 -- 172.21.15.67:0/400864170 wait complete. 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.473+0000 7f454d5db700 1 Processor -- start 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.473+0000 7f454d5db700 1 -- start start 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.473+0000 7f454d5db700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4548101a90 0x7f4548102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.473+0000 7f454d5db700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548102450 0x7f45481031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.473+0000 7f454d5db700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4548109520 0x7f45481036f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.473+0000 7f454d5db700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f45481136b0 con 0x7f4548102450 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.473+0000 7f454d5db700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4548113530 con 0x7f4548101a90 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.473+0000 7f454d5db700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4548113830 con 0x7f4548109520 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f45467fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548102450 0x7f45481031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f45467fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548102450 0x7f45481031b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39730/0 (socket says 172.21.15.67:39730) 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f45467fc700 1 -- 172.21.15.67:0/1231988294 learned_addr learned my addr 172.21.15.67:0/1231988294 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f4546ffd700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4548101a90 0x7f4548102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:17.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f45477fe700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4548109520 0x7f45481036f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f45467fc700 1 -- 172.21.15.67:0/1231988294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4548109520 msgr2=0x7f45481036f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f45467fc700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4548109520 0x7f45481036f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f45467fc700 1 -- 172.21.15.67:0/1231988294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4548101a90 msgr2=0x7f4548102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f45467fc700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4548101a90 0x7f4548102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f45467fc700 1 -- 172.21.15.67:0/1231988294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45481a8150 con 0x7f4548102450 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f4546ffd700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4548101a90 0x7f4548102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f45467fc700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548102450 0x7f45481031b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f453000d950 tx=0x7f453000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f453ffff700 1 -- 172.21.15.67:0/1231988294 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4530016070 con 0x7f4548102450 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f453ffff700 1 -- 172.21.15.67:0/1231988294 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4530010460 con 0x7f4548102450 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f454d5db700 1 -- 172.21.15.67:0/1231988294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45481a8440 con 0x7f4548102450 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f453ffff700 1 -- 172.21.15.67:0/1231988294 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f453000f620 con 0x7f4548102450 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.474+0000 7f454d5db700 1 -- 172.21.15.67:0/1231988294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45481a8980 con 0x7f4548102450 2024-05-03T17:21:17.477 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.475+0000 7f454d5db700 1 -- 172.21.15.67:0/1231988294 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f454810ba00 con 0x7f4548102450 2024-05-03T17:21:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.477+0000 7f453ffff700 1 -- 172.21.15.67:0/1231988294 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4530019070 con 0x7f4548102450 2024-05-03T17:21:17.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.478+0000 7f453ffff700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45340b2910 0x7f45340b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.478+0000 7f4546ffd700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45340b2910 0x7f45340b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.478+0000 7f453ffff700 1 -- 172.21.15.67:0/1231988294 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f45300e2020 con 0x7f4548102450 2024-05-03T17:21:17.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.479+0000 7f4546ffd700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45340b2910 0x7f45340b4dd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4538009e60 tx=0x7f4538006230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:17.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.482+0000 7f453ffff700 1 -- 172.21.15.67:0/1231988294 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f45300a9bd0 con 0x7f4548102450 2024-05-03T17:21:17.734 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.733+0000 7f454d5db700 1 -- 172.21.15.67:0/1231988294 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f45481041a0 con 0x7f45340b2910 2024-05-03T17:21:17.734 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:17 smithi067 ceph-mon[29731]: pgmap v316: 97 pgs: 97 active+clean; 602 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:17.738 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.738+0000 7f453ffff700 1 -- 172.21.15.67:0/1231988294 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f45481041a0 con 0x7f45340b2910 2024-05-03T17:21:17.739 INFO:teuthology.orchestra.run.smithi067.stdout:/volumes/qa/sv_1/32cff0e8-2085-4382-86b8-36aa14853e65 2024-05-03T17:21:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.740+0000 7f454d5db700 1 -- 172.21.15.67:0/1231988294 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45340b2910 msgr2=0x7f45340b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.740+0000 7f454d5db700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45340b2910 0x7f45340b4dd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4538009e60 tx=0x7f4538006230 comp rx=0 tx=0).stop 2024-05-03T17:21:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.740+0000 7f454d5db700 1 -- 172.21.15.67:0/1231988294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548102450 msgr2=0x7f45481031b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:17.741 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.740+0000 7f454d5db700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548102450 0x7f45481031b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f453000d950 tx=0x7f453000dd10 comp rx=0 tx=0).stop 2024-05-03T17:21:17.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.740+0000 7f454d5db700 1 -- 172.21.15.67:0/1231988294 shutdown_connections 2024-05-03T17:21:17.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.740+0000 7f454d5db700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f45340b2910 0x7f45340b4dd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:17.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.740+0000 7f454d5db700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4548109520 0x7f45481036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:17.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.740+0000 7f454d5db700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4548101a90 0x7f4548102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:17.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.740+0000 7f454d5db700 1 --2- 172.21.15.67:0/1231988294 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4548102450 0x7f45481031b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:17.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.740+0000 7f454d5db700 1 -- 172.21.15.67:0/1231988294 >> 172.21.15.67:0/1231988294 conn(0x7f4548078a10 msgr2=0x7f45481001c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:17.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.740+0000 7f454d5db700 1 -- 172.21.15.67:0/1231988294 shutdown_connections 2024-05-03T17:21:17.742 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:17.741+0000 7f454d5db700 1 -- 172.21.15.67:0/1231988294 wait complete. 2024-05-03T17:21:18.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:17 smithi082 ceph-mon[33535]: pgmap v316: 97 pgs: 97 active+clean; 602 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:18.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:17 smithi070 ceph-mon[33250]: pgmap v316: 97 pgs: 97 active+clean; 602 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.7 KiB/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:18.461 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:21:18.716 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:18.838 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:18 smithi067 ceph-mon[29731]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:19.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:18 smithi082 ceph-mon[33535]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:19.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:18 smithi070 ceph-mon[33250]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:21:19.899 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:19 smithi067 ceph-mon[29731]: pgmap v317: 97 pgs: 97 active+clean; 622 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.7 KiB/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:20.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:19 smithi082 ceph-mon[33535]: pgmap v317: 97 pgs: 97 active+clean; 622 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.7 KiB/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:20.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:19 smithi070 ceph-mon[33250]: pgmap v317: 97 pgs: 97 active+clean; 622 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 3.7 KiB/s wr, 7 op/s; 2 B/s, 0 objects/s recovering 2024-05-03T17:21:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.454+0000 7fefee671700 1 -- 172.21.15.67:0/2899580314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefe8108850 msgr2=0x7fefe81103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.454+0000 7fefee671700 1 --2- 172.21.15.67:0/2899580314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefe8108850 0x7fefe81103d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fefe000b0d0 tx=0x7fefe000b3e0 comp rx=0 tx=0).stop 2024-05-03T17:21:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.454+0000 7fefee671700 1 -- 172.21.15.67:0/2899580314 shutdown_connections 2024-05-03T17:21:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.454+0000 7fefee671700 1 --2- 172.21.15.67:0/2899580314 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefe8108850 0x7fefe81103d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.454+0000 7fefee671700 1 --2- 172.21.15.67:0/2899580314 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe8103750 0x7fefe8103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.454+0000 7fefee671700 1 --2- 172.21.15.67:0/2899580314 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fefe8104100 0x7fefe8108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.454+0000 7fefee671700 1 -- 172.21.15.67:0/2899580314 >> 172.21.15.67:0/2899580314 conn(0x7fefe807a600 msgr2=0x7fefe807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.454+0000 7fefee671700 1 -- 172.21.15.67:0/2899580314 shutdown_connections 2024-05-03T17:21:20.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.454+0000 7fefee671700 1 -- 172.21.15.67:0/2899580314 wait complete. 2024-05-03T17:21:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.454+0000 7fefee671700 1 Processor -- start 2024-05-03T17:21:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.454+0000 7fefee671700 1 -- start start 2024-05-03T17:21:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefee671700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefe8103750 0x7fefe819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefee671700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fefe8104100 0x7fefe819fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefee671700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe8108850 0x7fefe8199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefee671700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fefe8113620 con 0x7fefe8103750 2024-05-03T17:21:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefee671700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fefe81134a0 con 0x7fefe8108850 2024-05-03T17:21:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefee671700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fefe81137a0 con 0x7fefe8104100 2024-05-03T17:21:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefe7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefe8103750 0x7fefe819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefe7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefe8103750 0x7fefe819f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39758/0 (socket says 172.21.15.67:39758) 2024-05-03T17:21:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefe7fff700 1 -- 172.21.15.67:0/2793147073 learned_addr learned my addr 172.21.15.67:0/2793147073 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:20.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefe77fe700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fefe8104100 0x7fefe819fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefecc0e700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe8108850 0x7fefe8199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefe7fff700 1 -- 172.21.15.67:0/2793147073 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fefe8104100 msgr2=0x7fefe819fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefe7fff700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fefe8104100 0x7fefe819fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefe7fff700 1 -- 172.21.15.67:0/2793147073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe8108850 msgr2=0x7fefe8199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefe7fff700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe8108850 0x7fefe8199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefe7fff700 1 -- 172.21.15.67:0/2793147073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe8199fc0 con 0x7fefe8103750 2024-05-03T17:21:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.455+0000 7fefecc0e700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe8108850 0x7fefe8199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:21:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.456+0000 7fefe7fff700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefe8103750 0x7fefe819f6a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fefd800c920 tx=0x7fefd800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.456+0000 7fefe57fa700 1 -- 172.21.15.67:0/2793147073 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8007b00 con 0x7fefe8103750 2024-05-03T17:21:20.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.456+0000 7fefee671700 1 -- 172.21.15.67:0/2793147073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe819a2b0 con 0x7fefe8103750 2024-05-03T17:21:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.456+0000 7fefe57fa700 1 -- 172.21.15.67:0/2793147073 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fefd8004ce0 con 0x7fefe8103750 2024-05-03T17:21:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.456+0000 7fefe57fa700 1 -- 172.21.15.67:0/2793147073 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefd8005500 con 0x7fefe8103750 2024-05-03T17:21:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.456+0000 7fefee671700 1 -- 172.21.15.67:0/2793147073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe81a6570 con 0x7fefe8103750 2024-05-03T17:21:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.457+0000 7fefee671700 1 -- 172.21.15.67:0/2793147073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe806a190 con 0x7fefe8103750 2024-05-03T17:21:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.458+0000 7fefe57fa700 1 -- 172.21.15.67:0/2793147073 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fefd8007ca0 con 0x7fefe8103750 2024-05-03T17:21:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.459+0000 7fefe57fa700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fefd40b2910 0x7fefd40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.459+0000 7fefe77fe700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fefd40b2910 0x7fefd40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.459+0000 7fefe57fa700 1 -- 172.21.15.67:0/2793147073 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fefd80e23c0 con 0x7fefe8103750 2024-05-03T17:21:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.459+0000 7fefe77fe700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fefd40b2910 0x7fefd40b4dd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fefe8103000 tx=0x7fefd000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:20.468 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.463+0000 7fefe57fa700 1 -- 172.21.15.67:0/2793147073 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fefd8014030 con 0x7fefe8103750 2024-05-03T17:21:20.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.726+0000 7fefee671700 1 -- 172.21.15.67:0/2793147073 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefe819ad90 con 0x7fefe8103750 2024-05-03T17:21:20.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.727+0000 7fefe57fa700 1 -- 172.21.15.67:0/2793147073 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11850 (secure 0 0 0) 0x7fefd80a9f10 con 0x7fefe8103750 2024-05-03T17:21:20.728 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:21:20.728 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/316463706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":316463706},{"type":"v1","addr":"172.21.15.67:6839","nonce":316463706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3422086111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3422086111},{"type":"v1","addr":"172.21.15.67:6841","nonce":3422086111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T17:20:39.322107+0000","modified":"2024-05-03T17:21:01.753515+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24464,"mds_1":24446,"mds_2":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24421":{"gid":24421,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1568454842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1568454842},{"type":"v1","addr":"172.21.15.82:6835","nonce":1568454842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/370282857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":370282857},{"type":"v1","addr":"172.21.15.82:6837","nonce":370282857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.82:6839/2084218047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2084218047},{"type":"v1","addr":"172.21.15.82:6839","nonce":2084218047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.70:6837/3323713018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3323713018},{"type":"v1","addr":"172.21.15.70:6837","nonce":3323713018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6839/902276547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":902276547},{"type":"v1","addr":"172.21.15.70:6839","nonce":902276547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24446,24464,24442]},"id":1}]} 2024-05-03T17:21:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.729+0000 7fefee671700 1 -- 172.21.15.67:0/2793147073 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fefd40b2910 msgr2=0x7fefd40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.729+0000 7fefee671700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fefd40b2910 0x7fefd40b4dd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fefe8103000 tx=0x7fefd000b410 comp rx=0 tx=0).stop 2024-05-03T17:21:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.729+0000 7fefee671700 1 -- 172.21.15.67:0/2793147073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefe8103750 msgr2=0x7fefe819f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.729+0000 7fefee671700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefe8103750 0x7fefe819f6a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fefd800c920 tx=0x7fefd800cc30 comp rx=0 tx=0).stop 2024-05-03T17:21:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.729+0000 7fefee671700 1 -- 172.21.15.67:0/2793147073 shutdown_connections 2024-05-03T17:21:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.729+0000 7fefee671700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fefd40b2910 0x7fefd40b4dd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.729+0000 7fefee671700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fefe8108850 0x7fefe8199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.729+0000 7fefee671700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fefe8103750 0x7fefe819f6a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.729+0000 7fefee671700 1 --2- 172.21.15.67:0/2793147073 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fefe8104100 0x7fefe819fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.729+0000 7fefee671700 1 -- 172.21.15.67:0/2793147073 >> 172.21.15.67:0/2793147073 conn(0x7fefe807a600 msgr2=0x7fefe81015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:20.730 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.729+0000 7fefee671700 1 -- 172.21.15.67:0/2793147073 shutdown_connections 2024-05-03T17:21:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:20.729+0000 7fefee671700 1 -- 172.21.15.67:0/2793147073 wait complete. 2024-05-03T17:21:20.731 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T17:21:21.460 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T17:21:21.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:21 smithi067 ceph-mon[29731]: pgmap v318: 97 pgs: 97 active+clean; 622 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.3 KiB/s wr, 7 op/s; 1 B/s, 0 objects/s recovering 2024-05-03T17:21:21.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:21 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2793147073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:21:21.710 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:22.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:21 smithi082 ceph-mon[33535]: pgmap v318: 97 pgs: 97 active+clean; 622 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.3 KiB/s wr, 7 op/s; 1 B/s, 0 objects/s recovering 2024-05-03T17:21:22.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:21 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2793147073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:21:22.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:21 smithi070 ceph-mon[33250]: pgmap v318: 97 pgs: 97 active+clean; 622 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.3 KiB/s wr, 7 op/s; 1 B/s, 0 objects/s recovering 2024-05-03T17:21:22.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:21 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2793147073' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:21:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.474+0000 7f169742e700 1 -- 172.21.15.67:0/1173263156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1690108870 msgr2=0x7f16901103f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.474+0000 7f169742e700 1 --2- 172.21.15.67:0/1173263156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1690108870 0x7f16901103f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f168800b0d0 tx=0x7f168800b3e0 comp rx=0 tx=0).stop 2024-05-03T17:21:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.474+0000 7f169742e700 1 -- 172.21.15.67:0/1173263156 shutdown_connections 2024-05-03T17:21:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.474+0000 7f169742e700 1 --2- 172.21.15.67:0/1173263156 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1690108870 0x7f16901103f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.474+0000 7f169742e700 1 --2- 172.21.15.67:0/1173263156 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1690103780 0x7f1690103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:23.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.474+0000 7f169742e700 1 --2- 172.21.15.67:0/1173263156 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1690104130 0x7f1690108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.474+0000 7f169742e700 1 -- 172.21.15.67:0/1173263156 >> 172.21.15.67:0/1173263156 conn(0x7f169007a610 msgr2=0x7f169007aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.475+0000 7f169742e700 1 -- 172.21.15.67:0/1173263156 shutdown_connections 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.475+0000 7f169742e700 1 -- 172.21.15.67:0/1173263156 wait complete. 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.475+0000 7f169742e700 1 Processor -- start 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.475+0000 7f169742e700 1 -- start start 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.475+0000 7f169742e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1690103780 0x7f169019f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f169742e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1690104130 0x7f169019fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f169742e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1690108870 0x7f1690199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f169742e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1690113640 con 0x7f1690104130 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f169742e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f16901134c0 con 0x7f1690108870 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f169742e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f16901137c0 con 0x7f1690103780 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16949c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1690104130 0x7f169019fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16949c9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1690104130 0x7f169019fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39784/0 (socket says 172.21.15.67:39784) 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16949c9700 1 -- 172.21.15.67:0/3216352599 learned_addr learned my addr 172.21.15.67:0/3216352599 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:23.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16959cb700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1690108870 0x7f1690199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16951ca700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1690103780 0x7f169019f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16949c9700 1 -- 172.21.15.67:0/3216352599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1690103780 msgr2=0x7f169019f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16949c9700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1690103780 0x7f169019f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16949c9700 1 -- 172.21.15.67:0/3216352599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1690108870 msgr2=0x7f1690199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16949c9700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1690108870 0x7f1690199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16949c9700 1 -- 172.21.15.67:0/3216352599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1690199fe0 con 0x7f1690104130 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16951ca700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1690103780 0x7f169019f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16949c9700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1690104130 0x7f169019fc00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f168c00e9a0 tx=0x7f168c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16867fc700 1 -- 172.21.15.67:0/3216352599 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c00cbd0 con 0x7f1690104130 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16867fc700 1 -- 172.21.15.67:0/3216352599 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f168c00cd70 con 0x7f1690104130 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f169742e700 1 -- 172.21.15.67:0/3216352599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f169019a2d0 con 0x7f1690104130 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f16867fc700 1 -- 172.21.15.67:0/3216352599 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168c018680 con 0x7f1690104130 2024-05-03T17:21:23.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.476+0000 7f169742e700 1 -- 172.21.15.67:0/3216352599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16901a6590 con 0x7f1690104130 2024-05-03T17:21:23.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.477+0000 7f169742e700 1 -- 172.21.15.67:0/3216352599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f169006a190 con 0x7f1690104130 2024-05-03T17:21:23.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.479+0000 7f16867fc700 1 -- 172.21.15.67:0/3216352599 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f168c010430 con 0x7f1690104130 2024-05-03T17:21:23.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.479+0000 7f16867fc700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f167c0b2990 0x7f167c0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:23.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.480+0000 7f16951ca700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f167c0b2990 0x7f167c0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:23.481 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.480+0000 7f16867fc700 1 -- 172.21.15.67:0/3216352599 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f168c0e2a60 con 0x7f1690104130 2024-05-03T17:21:23.482 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.480+0000 7f16951ca700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f167c0b2990 0x7f167c0b4e50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1680005950 tx=0x7f16800058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:23.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.483+0000 7f16867fc700 1 -- 172.21.15.67:0/3216352599 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f168c0aa610 con 0x7f1690104130 2024-05-03T17:21:23.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.703+0000 7f169742e700 1 -- 172.21.15.67:0/3216352599 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f169002e510 con 0x7f1690104130 2024-05-03T17:21:23.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.704+0000 7f16867fc700 1 -- 172.21.15.67:0/3216352599 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22635 (secure 0 0 0) 0x7f168c0a9f10 con 0x7f1690104130 2024-05-03T17:21:23.706 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:21:23.707 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":81,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","created":"2024-05-03T17:12:11.709381+0000","modified":"2024-05-03T17:21:07.584731+0000","last_up_change":"2024-05-03T17:19:29.310797+0000","last_in_change":"2024-05-03T17:19:08.848654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T17:15:36.413266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T17:20:31.546843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T17:20:35.261082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80962eb1-a506-49bb-a528-dee035eb4322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6803","nonce":1754365741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6805","nonce":1754365741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6809","nonce":1754365741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6807","nonce":1754365741}]},"public_addr":"172.21.15.67:6803/1754365741","cluster_addr":"172.21.15.67:6805/1754365741","heartbeat_back_addr":"172.21.15.67:6809/1754365741","heartbeat_front_addr":"172.21.15.67:6807/1754365741","state":["exists","up"]},{"osd":1,"uuid":"d7721d5f-4373-4cb2-bb57-bc368d6bb818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6801","nonce":1006339176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6803","nonce":1006339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6807","nonce":1006339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6805","nonce":1006339176}]},"public_addr":"172.21.15.70:6801/1006339176","cluster_addr":"172.21.15.70:6803/1006339176","heartbeat_back_addr":"172.21.15.70:6807/1006339176","heartbeat_front_addr":"172.21.15.70:6805/1006339176","state":["exists","up"]},{"osd":2,"uuid":"9992fd98-821b-40ce-b051-89befbfd87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6801","nonce":1888803264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6803","nonce":1888803264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6807","nonce":1888803264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6805","nonce":1888803264}]},"public_addr":"172.21.15.82:6801/1888803264","cluster_addr":"172.21.15.82:6803/1888803264","heartbeat_back_addr":"172.21.15.82:6807/1888803264","heartbeat_front_addr":"172.21.15.82:6805/1888803264","state":["exists","up"]},{"osd":3,"uuid":"14bfe6a6-0bc8-4766-8d36-fbaecc61a396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6811","nonce":313717723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6813","nonce":313717723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6817","nonce":313717723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6815","nonce":313717723}]},"public_addr":"172.21.15.67:6811/313717723","cluster_addr":"172.21.15.67:6813/313717723","heartbeat_back_addr":"172.21.15.67:6817/313717723","heartbeat_front_addr":"172.21.15.67:6815/313717723","state":["exists","up"]},{"osd":4,"uuid":"5a44aeb8-aabf-4824-be65-2c7170dbee52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6809","nonce":3526300828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6811","nonce":3526300828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6815","nonce":3526300828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6813","nonce":3526300828}]},"public_addr":"172.21.15.70:6809/3526300828","cluster_addr":"172.21.15.70:6811/3526300828","heartbeat_back_addr":"172.21.15.70:6815/3526300828","heartbeat_front_addr":"172.21.15.70:6813/3526300828","state":["exists","up"]},{"osd":5,"uuid":"a6ac56df-466e-4096-8fbf-835f5b544297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6809","nonce":3754223290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6811","nonce":3754223290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6815","nonce":3754223290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6813","nonce":3754223290}]},"public_addr":"172.21.15.82:6809/3754223290","cluster_addr":"172.21.15.82:6811/3754223290","heartbeat_back_addr":"172.21.15.82:6815/3754223290","heartbeat_front_addr":"172.21.15.82:6813/3754223290","state":["exists","up"]},{"osd":6,"uuid":"4d8a5e8a-66ac-462e-b960-24c0ee248fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6819","nonce":4033769161}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6821","nonce":4033769161}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6825","nonce":4033769161}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6823","nonce":4033769161}]},"public_addr":"172.21.15.67:6819/4033769161","cluster_addr":"172.21.15.67:6821/4033769161","heartbeat_back_addr":"172.21.15.67:6825/4033769161","heartbeat_front_addr":"172.21.15.67:6823/4033769161","state":["exists","up"]},{"osd":7,"uuid":"eb983837-8172-4802-9843-6360cda8f4be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6817","nonce":3366146173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6819","nonce":3366146173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6823","nonce":3366146173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6821","nonce":3366146173}]},"public_addr":"172.21.15.70:6817/3366146173","cluster_addr":"172.21.15.70:6819/3366146173","heartbeat_back_addr":"172.21.15.70:6823/3366146173","heartbeat_front_addr":"172.21.15.70:6821/3366146173","state":["exists","up"]},{"osd":8,"uuid":"c9d4ce56-e940-48d1-a9b5-437139e3fed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6817","nonce":347699286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6819","nonce":347699286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6823","nonce":347699286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6821","nonce":347699286}]},"public_addr":"172.21.15.82:6817/347699286","cluster_addr":"172.21.15.82:6819/347699286","heartbeat_back_addr":"172.21.15.82:6823/347699286","heartbeat_front_addr":"172.21.15.82:6821/347699286","state":["exists","up"]},{"osd":9,"uuid":"09596d19-f2d7-485e-a028-9ce5468c6651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6827","nonce":2600095198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6829","nonce":2600095198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6833","nonce":2600095198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6831","nonce":2600095198}]},"public_addr":"172.21.15.67:6827/2600095198","cluster_addr":"172.21.15.67:6829/2600095198","heartbeat_back_addr":"172.21.15.67:6833/2600095198","heartbeat_front_addr":"172.21.15.67:6831/2600095198","state":["exists","up"]},{"osd":10,"uuid":"ec4cffbc-0127-47e5-8176-0972f74a5514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6825","nonce":2897583629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6827","nonce":2897583629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6831","nonce":2897583629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6829","nonce":2897583629}]},"public_addr":"172.21.15.70:6825/2897583629","cluster_addr":"172.21.15.70:6827/2897583629","heartbeat_back_addr":"172.21.15.70:6831/2897583629","heartbeat_front_addr":"172.21.15.70:6829/2897583629","state":["exists","up"]},{"osd":11,"uuid":"a8acfb0b-98e8-49e1-a114-f33aa8160b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6825","nonce":4082656149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6827","nonce":4082656149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6831","nonce":4082656149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6829","nonce":4082656149}]},"public_addr":"172.21.15.82:6825/4082656149","cluster_addr":"172.21.15.82:6827/4082656149","heartbeat_back_addr":"172.21.15.82:6831/4082656149","heartbeat_front_addr":"172.21.15.82:6829/4082656149","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:14:45.596665+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:09.538408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:33.608945+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:59.803102+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:25.187966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:49.892160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:16.575973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:42.115955+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:07.655558+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:35.356779+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:01.144695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:27.577513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/2131289004":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/3335149382":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/992151181":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2614686067":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/3137329076":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2323686292":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/2579128777":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/3335149382":"2024-05-04T17:13:06.324002+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T17:21:23.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.707+0000 7f169742e700 1 -- 172.21.15.67:0/3216352599 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f167c0b2990 msgr2=0x7f167c0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:23.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.707+0000 7f169742e700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f167c0b2990 0x7f167c0b4e50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1680005950 tx=0x7f16800058e0 comp rx=0 tx=0).stop 2024-05-03T17:21:23.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.707+0000 7f169742e700 1 -- 172.21.15.67:0/3216352599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1690104130 msgr2=0x7f169019fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:23.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.707+0000 7f169742e700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1690104130 0x7f169019fc00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f168c00e9a0 tx=0x7f168c00ecb0 comp rx=0 tx=0).stop 2024-05-03T17:21:23.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.707+0000 7f169742e700 1 -- 172.21.15.67:0/3216352599 shutdown_connections 2024-05-03T17:21:23.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.707+0000 7f169742e700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f167c0b2990 0x7f167c0b4e50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:23.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.707+0000 7f169742e700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1690108870 0x7f1690199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:23.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.707+0000 7f169742e700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1690103780 0x7f169019f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:23.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.707+0000 7f169742e700 1 --2- 172.21.15.67:0/3216352599 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1690104130 0x7f169019fc00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:23.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.707+0000 7f169742e700 1 -- 172.21.15.67:0/3216352599 >> 172.21.15.67:0/3216352599 conn(0x7f169007a610 msgr2=0x7f1690101620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:23.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.707+0000 7f169742e700 1 -- 172.21.15.67:0/3216352599 shutdown_connections 2024-05-03T17:21:23.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:23.707+0000 7f169742e700 1 -- 172.21.15.67:0/3216352599 wait complete. 2024-05-03T17:21:23.710 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:23 smithi067 ceph-mon[29731]: pgmap v319: 97 pgs: 97 active+clean; 622 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.3 KiB/s wr, 7 op/s 2024-05-03T17:21:24.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:23 smithi082 ceph-mon[33535]: pgmap v319: 97 pgs: 97 active+clean; 622 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.3 KiB/s wr, 7 op/s 2024-05-03T17:21:24.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:23 smithi070 ceph-mon[33250]: pgmap v319: 97 pgs: 97 active+clean; 622 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.3 KiB/s wr, 7 op/s 2024-05-03T17:21:24.422 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-03T17:21:24.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-03T17:21:24.678 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:24.929 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:24 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3216352599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:21:25.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:24 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3216352599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:21:25.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:24 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3216352599' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:21:25.872 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:25 smithi067 ceph-mon[29731]: pgmap v320: 97 pgs: 97 active+clean; 630 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-05-03T17:21:26.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:25 smithi082 ceph-mon[33535]: pgmap v320: 97 pgs: 97 active+clean; 630 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-05-03T17:21:26.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:25 smithi070 ceph-mon[33250]: pgmap v320: 97 pgs: 97 active+clean; 630 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 4.2 KiB/s wr, 9 op/s 2024-05-03T17:21:26.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.419+0000 7f6fb3359700 1 -- 172.21.15.67:0/347784161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac10adc0 msgr2=0x7f6fac1160b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.419+0000 7f6fb3359700 1 --2- 172.21.15.67:0/347784161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac10adc0 0x7f6fac1160b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c009990 tx=0x7f6f9c009ca0 comp rx=0 tx=0).stop 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.419+0000 7f6fb3359700 1 -- 172.21.15.67:0/347784161 shutdown_connections 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.419+0000 7f6fb3359700 1 --2- 172.21.15.67:0/347784161 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fac1165f0 0x7f6fac1189e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.419+0000 7f6fb3359700 1 --2- 172.21.15.67:0/347784161 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fac10a4a0 0x7f6fac10a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.419+0000 7f6fb3359700 1 --2- 172.21.15.67:0/347784161 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac10adc0 0x7f6fac1160b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.419+0000 7f6fb3359700 1 -- 172.21.15.67:0/347784161 >> 172.21.15.67:0/347784161 conn(0x7f6fac06e610 msgr2=0x7f6fac06ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.419+0000 7f6fb3359700 1 -- 172.21.15.67:0/347784161 shutdown_connections 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.419+0000 7f6fb3359700 1 -- 172.21.15.67:0/347784161 wait complete. 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb3359700 1 Processor -- start 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb3359700 1 -- start start 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb3359700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac10a4a0 0x7f6fac1a2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb3359700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fac10adc0 0x7f6fac1a28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb3359700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fac1165f0 0x7f6fac1a6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb3359700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac11bec0 con 0x7f6fac10a4a0 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb3359700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac11bd40 con 0x7f6fac10adc0 2024-05-03T17:21:26.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb3359700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f6fac11c040 con 0x7f6fac1165f0 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb10f5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac10a4a0 0x7f6fac1a2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb10f5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac10a4a0 0x7f6fac1a2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39806/0 (socket says 172.21.15.67:39806) 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb10f5700 1 -- 172.21.15.67:0/582245080 learned_addr learned my addr 172.21.15.67:0/582245080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb08f4700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fac10adc0 0x7f6fac1a28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb18f6700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fac1165f0 0x7f6fac1a6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb10f5700 1 -- 172.21.15.67:0/582245080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fac1165f0 msgr2=0x7f6fac1a6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb10f5700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fac1165f0 0x7f6fac1a6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb10f5700 1 -- 172.21.15.67:0/582245080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fac10adc0 msgr2=0x7f6fac1a28d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb10f5700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fac10adc0 0x7f6fac1a28d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.420+0000 7f6fb10f5700 1 -- 172.21.15.67:0/582245080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fac1a73a0 con 0x7f6fac10a4a0 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.421+0000 7f6fb10f5700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac10a4a0 0x7f6fac1a2390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800d950 tx=0x7f6fa800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.421+0000 7f6fa27fc700 1 -- 172.21.15.67:0/582245080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8016070 con 0x7f6fac10a4a0 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.421+0000 7f6fa27fc700 1 -- 172.21.15.67:0/582245080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6fa8009df0 con 0x7f6fac10a4a0 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.421+0000 7f6fb3359700 1 -- 172.21.15.67:0/582245080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fac1a7690 con 0x7f6fac10a4a0 2024-05-03T17:21:26.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.421+0000 7f6fa27fc700 1 -- 172.21.15.67:0/582245080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa800f3c0 con 0x7f6fac10a4a0 2024-05-03T17:21:26.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.421+0000 7f6fb3359700 1 -- 172.21.15.67:0/582245080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fac1aef10 con 0x7f6fac10a4a0 2024-05-03T17:21:26.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.422+0000 7f6fb3359700 1 -- 172.21.15.67:0/582245080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fac06fba0 con 0x7f6fac10a4a0 2024-05-03T17:21:26.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.424+0000 7f6fa27fc700 1 -- 172.21.15.67:0/582245080 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f6fa8019070 con 0x7f6fac10a4a0 2024-05-03T17:21:26.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.425+0000 7f6fa27fc700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f6f980b2910 0x7f6f980b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:26.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.425+0000 7f6fb08f4700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f6f980b2910 0x7f6f980b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:26.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.425+0000 7f6fa27fc700 1 -- 172.21.15.67:0/582245080 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f6fa80e1f40 con 0x7f6fac10a4a0 2024-05-03T17:21:26.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.425+0000 7f6fb08f4700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f6f980b2910 0x7f6f980b4dd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00b2e0 tx=0x7f6f9c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:26.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.429+0000 7f6fa27fc700 1 -- 172.21.15.67:0/582245080 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6fa80a9af0 con 0x7f6fac10a4a0 2024-05-03T17:21:26.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.636+0000 7f6fb3359700 1 -- 172.21.15.67:0/582245080 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6fac0638e0 con 0x7f6f980b2910 2024-05-03T17:21:26.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.644+0000 7f6fa27fc700 1 -- 172.21.15.67:0/582245080 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258915 (secure 0 0 0) 0x7f6f9c067380 con 0x7f6f980b2910 2024-05-03T17:21:26.649 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.647+0000 7f6fb3359700 1 -- 172.21.15.67:0/582245080 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f6f980b2910 msgr2=0x7f6f980b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.647+0000 7f6fb3359700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f6f980b2910 0x7f6f980b4dd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6f9c00b2e0 tx=0x7f6f9c005960 comp rx=0 tx=0).stop 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.647+0000 7f6fb3359700 1 -- 172.21.15.67:0/582245080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac10a4a0 msgr2=0x7f6fac1a2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.647+0000 7f6fb3359700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac10a4a0 0x7f6fac1a2390 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6fa800d950 tx=0x7f6fa800dc60 comp rx=0 tx=0).stop 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.647+0000 7f6fb3359700 1 -- 172.21.15.67:0/582245080 shutdown_connections 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.647+0000 7f6fb3359700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f6f980b2910 0x7f6f980b4dd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.647+0000 7f6fb3359700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6fac1165f0 0x7f6fac1a6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.647+0000 7f6fb3359700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6fac10a4a0 0x7f6fac1a2390 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.647+0000 7f6fb3359700 1 --2- 172.21.15.67:0/582245080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6fac10adc0 0x7f6fac1a28d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.647+0000 7f6fb3359700 1 -- 172.21.15.67:0/582245080 >> 172.21.15.67:0/582245080 conn(0x7f6fac06e610 msgr2=0x7f6fac072220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.647+0000 7f6fb3359700 1 -- 172.21.15.67:0/582245080 shutdown_connections 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:26.647+0000 7f6fb3359700 1 -- 172.21.15.67:0/582245080 wait complete. 2024-05-03T17:21:26.652 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-03T17:21:27.386 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-05-03T17:21:26.428461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":645298,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":246,"num_read_kb":206,"num_write":282,"num_write_kb":2250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":228,"ondisk_log_size":228,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":359904,"kb_used_data":10728,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581920,"statfs":{"total":1151940427776,"available":1151571886080,"internally_reserved":0,"allocated":10985472,"data_stored":7069552,"data_compressed":178158,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28482,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":63,"num_read_kb":59,"num_write":20,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003039"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595780+0000","last_change":"2024-05-03T17:20:37.204469+0000","last_active":"2024-05-03T17:21:07.595780+0000","last_peered":"2024-05-03T17:21:07.595780+0000","last_clean":"2024-05-03T17:21:07.595780+0000","last_became_active":"2024-05-03T17:20:37.204098+0000","last_became_peered":"2024-05-03T17:20:37.204098+0000","last_unstale":"2024-05-03T17:21:07.595780+0000","last_undegraded":"2024-05-03T17:21:07.595780+0000","last_fullsized":"2024-05-03T17:21:07.595780+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:03:29.875319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599098+0000","last_change":"2024-05-03T17:20:37.201171+0000","last_active":"2024-05-03T17:21:07.599098+0000","last_peered":"2024-05-03T17:21:07.599098+0000","last_clean":"2024-05-03T17:21:07.599098+0000","last_became_active":"2024-05-03T17:20:37.200852+0000","last_became_peered":"2024-05-03T17:20:37.200852+0000","last_unstale":"2024-05-03T17:21:07.599098+0000","last_undegraded":"2024-05-03T17:21:07.599098+0000","last_fullsized":"2024-05-03T17:21:07.599098+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:12:06.549919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598922+0000","last_change":"2024-05-03T17:20:37.201925+0000","last_active":"2024-05-03T17:21:07.598922+0000","last_peered":"2024-05-03T17:21:07.598922+0000","last_clean":"2024-05-03T17:21:07.598922+0000","last_became_active":"2024-05-03T17:20:37.201619+0000","last_became_peered":"2024-05-03T17:20:37.201619+0000","last_unstale":"2024-05-03T17:21:07.598922+0000","last_undegraded":"2024-05-03T17:21:07.598922+0000","last_fullsized":"2024-05-03T17:21:07.598922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:50:19.019887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603123+0000","last_change":"2024-05-03T17:20:37.203535+0000","last_active":"2024-05-03T17:21:07.603123+0000","last_peered":"2024-05-03T17:21:07.603123+0000","last_clean":"2024-05-03T17:21:07.603123+0000","last_became_active":"2024-05-03T17:20:37.203175+0000","last_became_peered":"2024-05-03T17:20:37.203175+0000","last_unstale":"2024-05-03T17:21:07.603123+0000","last_undegraded":"2024-05-03T17:21:07.603123+0000","last_fullsized":"2024-05-03T17:21:07.603123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:59:41.026709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597618+0000","last_change":"2024-05-03T17:20:37.200283+0000","last_active":"2024-05-03T17:21:07.597618+0000","last_peered":"2024-05-03T17:21:07.597618+0000","last_clean":"2024-05-03T17:21:07.597618+0000","last_became_active":"2024-05-03T17:20:37.199105+0000","last_became_peered":"2024-05-03T17:20:37.199105+0000","last_unstale":"2024-05-03T17:21:07.597618+0000","last_undegraded":"2024-05-03T17:21:07.597618+0000","last_fullsized":"2024-05-03T17:21:07.597618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:56:39.990497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597382+0000","last_change":"2024-05-03T17:21:07.597382+0000","last_active":"2024-05-03T17:21:07.597382+0000","last_peered":"2024-05-03T17:21:07.597382+0000","last_clean":"2024-05-03T17:21:07.597382+0000","last_became_active":"2024-05-03T17:21:07.596935+0000","last_became_peered":"2024-05-03T17:21:07.596935+0000","last_unstale":"2024-05-03T17:21:07.597382+0000","last_undegraded":"2024-05-03T17:21:07.597382+0000","last_fullsized":"2024-05-03T17:21:07.597382+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T19:30:50.200783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596543+0000","last_change":"2024-05-03T17:20:37.211378+0000","last_active":"2024-05-03T17:21:07.596543+0000","last_peered":"2024-05-03T17:21:07.596543+0000","last_clean":"2024-05-03T17:21:07.596543+0000","last_became_active":"2024-05-03T17:20:37.210684+0000","last_became_peered":"2024-05-03T17:20:37.210684+0000","last_unstale":"2024-05-03T17:21:07.596543+0000","last_undegraded":"2024-05-03T17:21:07.596543+0000","last_fullsized":"2024-05-03T17:21:07.596543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:35:24.737186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592032+0000","last_change":"2024-05-03T17:20:37.202989+0000","last_active":"2024-05-03T17:21:07.592032+0000","last_peered":"2024-05-03T17:21:07.592032+0000","last_clean":"2024-05-03T17:21:07.592032+0000","last_became_active":"2024-05-03T17:20:37.199975+0000","last_became_peered":"2024-05-03T17:20:37.199975+0000","last_unstale":"2024-05-03T17:21:07.592032+0000","last_undegraded":"2024-05-03T17:21:07.592032+0000","last_fullsized":"2024-05-03T17:21:07.592032+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:13:43.875472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598772+0000","last_change":"2024-05-03T17:20:37.206754+0000","last_active":"2024-05-03T17:21:07.598772+0000","last_peered":"2024-05-03T17:21:07.598772+0000","last_clean":"2024-05-03T17:21:07.598772+0000","last_became_active":"2024-05-03T17:20:37.206423+0000","last_became_peered":"2024-05-03T17:20:37.206423+0000","last_unstale":"2024-05-03T17:21:07.598772+0000","last_undegraded":"2024-05-03T17:21:07.598772+0000","last_fullsized":"2024-05-03T17:21:07.598772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:10:30.227969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593231+0000","last_change":"2024-05-03T17:20:37.211868+0000","last_active":"2024-05-03T17:21:07.593231+0000","last_peered":"2024-05-03T17:21:07.593231+0000","last_clean":"2024-05-03T17:21:07.593231+0000","last_became_active":"2024-05-03T17:20:37.211486+0000","last_became_peered":"2024-05-03T17:20:37.211486+0000","last_unstale":"2024-05-03T17:21:07.593231+0000","last_undegraded":"2024-05-03T17:21:07.593231+0000","last_fullsized":"2024-05-03T17:21:07.593231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:41:03.966446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596847+0000","last_change":"2024-05-03T17:20:37.206511+0000","last_active":"2024-05-03T17:21:07.596847+0000","last_peered":"2024-05-03T17:21:07.596847+0000","last_clean":"2024-05-03T17:21:07.596847+0000","last_became_active":"2024-05-03T17:20:37.206180+0000","last_became_peered":"2024-05-03T17:20:37.206180+0000","last_unstale":"2024-05-03T17:21:07.596847+0000","last_undegraded":"2024-05-03T17:21:07.596847+0000","last_fullsized":"2024-05-03T17:21:07.596847+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:59:07.277623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603068+0000","last_change":"2024-05-03T17:20:37.204842+0000","last_active":"2024-05-03T17:21:07.603068+0000","last_peered":"2024-05-03T17:21:07.603068+0000","last_clean":"2024-05-03T17:21:07.603068+0000","last_became_active":"2024-05-03T17:20:37.204275+0000","last_became_peered":"2024-05-03T17:20:37.204275+0000","last_unstale":"2024-05-03T17:21:07.603068+0000","last_undegraded":"2024-05-03T17:21:07.603068+0000","last_fullsized":"2024-05-03T17:21:07.603068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:27:00.021413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597215+0000","last_change":"2024-05-03T17:21:07.597215+0000","last_active":"2024-05-03T17:21:07.597215+0000","last_peered":"2024-05-03T17:21:07.597215+0000","last_clean":"2024-05-03T17:21:07.597215+0000","last_became_active":"2024-05-03T17:21:07.596833+0000","last_became_peered":"2024-05-03T17:21:07.596833+0000","last_unstale":"2024-05-03T17:21:07.597215+0000","last_undegraded":"2024-05-03T17:21:07.597215+0000","last_fullsized":"2024-05-03T17:21:07.597215+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:09:41.399761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597704+0000","last_change":"2024-05-03T17:20:37.200759+0000","last_active":"2024-05-03T17:21:07.597704+0000","last_peered":"2024-05-03T17:21:07.597704+0000","last_clean":"2024-05-03T17:21:07.597704+0000","last_became_active":"2024-05-03T17:20:37.200410+0000","last_became_peered":"2024-05-03T17:20:37.200410+0000","last_unstale":"2024-05-03T17:21:07.597704+0000","last_undegraded":"2024-05-03T17:21:07.597704+0000","last_fullsized":"2024-05-03T17:21:07.597704+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:51:03.719792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597446+0000","last_change":"2024-05-03T17:20:37.204534+0000","last_active":"2024-05-03T17:21:07.597446+0000","last_peered":"2024-05-03T17:21:07.597446+0000","last_clean":"2024-05-03T17:21:07.597446+0000","last_became_active":"2024-05-03T17:20:37.204167+0000","last_became_peered":"2024-05-03T17:20:37.204167+0000","last_unstale":"2024-05-03T17:21:07.597446+0000","last_undegraded":"2024-05-03T17:21:07.597446+0000","last_fullsized":"2024-05-03T17:21:07.597446+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:31:35.730465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598463+0000","last_change":"2024-05-03T17:20:37.207658+0000","last_active":"2024-05-03T17:21:07.598463+0000","last_peered":"2024-05-03T17:21:07.598463+0000","last_clean":"2024-05-03T17:21:07.598463+0000","last_became_active":"2024-05-03T17:20:37.207313+0000","last_became_peered":"2024-05-03T17:20:37.207313+0000","last_unstale":"2024-05-03T17:21:07.598463+0000","last_undegraded":"2024-05-03T17:21:07.598463+0000","last_fullsized":"2024-05-03T17:21:07.598463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:47:21.287559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595876+0000","last_change":"2024-05-03T17:20:37.205021+0000","last_active":"2024-05-03T17:21:07.595876+0000","last_peered":"2024-05-03T17:21:07.595876+0000","last_clean":"2024-05-03T17:21:07.595876+0000","last_became_active":"2024-05-03T17:20:37.204630+0000","last_became_peered":"2024-05-03T17:20:37.204630+0000","last_unstale":"2024-05-03T17:21:07.595876+0000","last_undegraded":"2024-05-03T17:21:07.595876+0000","last_fullsized":"2024-05-03T17:21:07.595876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:03:00.318648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593860+0000","last_change":"2024-05-03T17:20:37.201829+0000","last_active":"2024-05-03T17:21:07.593860+0000","last_peered":"2024-05-03T17:21:07.593860+0000","last_clean":"2024-05-03T17:21:07.593860+0000","last_became_active":"2024-05-03T17:20:37.201465+0000","last_became_peered":"2024-05-03T17:20:37.201465+0000","last_unstale":"2024-05-03T17:21:07.593860+0000","last_undegraded":"2024-05-03T17:21:07.593860+0000","last_fullsized":"2024-05-03T17:21:07.593860+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:05:46.844150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594023+0000","last_change":"2024-05-03T17:20:37.199138+0000","last_active":"2024-05-03T17:21:07.594023+0000","last_peered":"2024-05-03T17:21:07.594023+0000","last_clean":"2024-05-03T17:21:07.594023+0000","last_became_active":"2024-05-03T17:20:37.198722+0000","last_became_peered":"2024-05-03T17:20:37.198722+0000","last_unstale":"2024-05-03T17:21:07.594023+0000","last_undegraded":"2024-05-03T17:21:07.594023+0000","last_fullsized":"2024-05-03T17:21:07.594023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:12:04.764356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593787+0000","last_change":"2024-05-03T17:20:33.168111+0000","last_active":"2024-05-03T17:21:07.593787+0000","last_peered":"2024-05-03T17:21:07.593787+0000","last_clean":"2024-05-03T17:21:07.593787+0000","last_became_active":"2024-05-03T17:20:33.167892+0000","last_became_peered":"2024-05-03T17:20:33.167892+0000","last_unstale":"2024-05-03T17:21:07.593787+0000","last_undegraded":"2024-05-03T17:21:07.593787+0000","last_fullsized":"2024-05-03T17:21:07.593787+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:55:45.235287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592219+0000","last_change":"2024-05-03T17:20:37.208673+0000","last_active":"2024-05-03T17:21:07.592219+0000","last_peered":"2024-05-03T17:21:07.592219+0000","last_clean":"2024-05-03T17:21:07.592219+0000","last_became_active":"2024-05-03T17:20:37.208360+0000","last_became_peered":"2024-05-03T17:20:37.208360+0000","last_unstale":"2024-05-03T17:21:07.592219+0000","last_undegraded":"2024-05-03T17:21:07.592219+0000","last_fullsized":"2024-05-03T17:21:07.592219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:42:02.966225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592593+0000","last_change":"2024-05-03T17:20:33.284052+0000","last_active":"2024-05-03T17:21:07.592593+0000","last_peered":"2024-05-03T17:21:07.592593+0000","last_clean":"2024-05-03T17:21:07.592593+0000","last_became_active":"2024-05-03T17:20:33.283635+0000","last_became_peered":"2024-05-03T17:20:33.283635+0000","last_unstale":"2024-05-03T17:21:07.592593+0000","last_undegraded":"2024-05-03T17:21:07.592593+0000","last_fullsized":"2024-05-03T17:21:07.592593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:17:51.219359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598804+0000","last_change":"2024-05-03T17:20:37.197321+0000","last_active":"2024-05-03T17:21:07.598804+0000","last_peered":"2024-05-03T17:21:07.598804+0000","last_clean":"2024-05-03T17:21:07.598804+0000","last_became_active":"2024-05-03T17:20:37.196569+0000","last_became_peered":"2024-05-03T17:20:37.196569+0000","last_unstale":"2024-05-03T17:21:07.598804+0000","last_undegraded":"2024-05-03T17:21:07.598804+0000","last_fullsized":"2024-05-03T17:21:07.598804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:38:03.274058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598373+0000","last_change":"2024-05-03T17:20:33.168828+0000","last_active":"2024-05-03T17:21:07.598373+0000","last_peered":"2024-05-03T17:21:07.598373+0000","last_clean":"2024-05-03T17:21:07.598373+0000","last_became_active":"2024-05-03T17:20:33.168408+0000","last_became_peered":"2024-05-03T17:20:33.168408+0000","last_unstale":"2024-05-03T17:21:07.598373+0000","last_undegraded":"2024-05-03T17:21:07.598373+0000","last_fullsized":"2024-05-03T17:21:07.598373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:50:57.790353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598587+0000","last_change":"2024-05-03T17:20:37.199853+0000","last_active":"2024-05-03T17:21:07.598587+0000","last_peered":"2024-05-03T17:21:07.598587+0000","last_clean":"2024-05-03T17:21:07.598587+0000","last_became_active":"2024-05-03T17:20:37.199526+0000","last_became_peered":"2024-05-03T17:20:37.199526+0000","last_unstale":"2024-05-03T17:21:07.598587+0000","last_undegraded":"2024-05-03T17:21:07.598587+0000","last_fullsized":"2024-05-03T17:21:07.598587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:42:45.932795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597857+0000","last_change":"2024-05-03T17:20:33.168050+0000","last_active":"2024-05-03T17:21:07.597857+0000","last_peered":"2024-05-03T17:21:07.597857+0000","last_clean":"2024-05-03T17:21:07.597857+0000","last_became_active":"2024-05-03T17:20:33.167479+0000","last_became_peered":"2024-05-03T17:20:33.167479+0000","last_unstale":"2024-05-03T17:21:07.597857+0000","last_undegraded":"2024-05-03T17:21:07.597857+0000","last_fullsized":"2024-05-03T17:21:07.597857+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:39:32.626940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604399+0000","last_change":"2024-05-03T17:21:07.603996+0000","last_active":"2024-05-03T17:21:07.604399+0000","last_peered":"2024-05-03T17:21:07.604399+0000","last_clean":"2024-05-03T17:21:07.604399+0000","last_became_active":"2024-05-03T17:21:07.603647+0000","last_became_peered":"2024-05-03T17:21:07.603647+0000","last_unstale":"2024-05-03T17:21:07.604399+0000","last_undegraded":"2024-05-03T17:21:07.604399+0000","last_fullsized":"2024-05-03T17:21:07.604399+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:31:07.956642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"79'6","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:20.622143+0000","last_change":"2024-05-03T17:20:33.166560+0000","last_active":"2024-05-03T17:21:20.622143+0000","last_peered":"2024-05-03T17:21:20.622143+0000","last_clean":"2024-05-03T17:21:20.622143+0000","last_became_active":"2024-05-03T17:20:33.166136+0000","last_became_peered":"2024-05-03T17:20:33.166136+0000","last_unstale":"2024-05-03T17:21:20.622143+0000","last_undegraded":"2024-05-03T17:21:20.622143+0000","last_fullsized":"2024-05-03T17:21:20.622143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:38:54.132364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597525+0000","last_change":"2024-05-03T17:20:37.204336+0000","last_active":"2024-05-03T17:21:07.597525+0000","last_peered":"2024-05-03T17:21:07.597525+0000","last_clean":"2024-05-03T17:21:07.597525+0000","last_became_active":"2024-05-03T17:20:37.203962+0000","last_became_peered":"2024-05-03T17:20:37.203962+0000","last_unstale":"2024-05-03T17:21:07.597525+0000","last_undegraded":"2024-05-03T17:21:07.597525+0000","last_fullsized":"2024-05-03T17:21:07.597525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:29:48.081362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592586+0000","last_change":"2024-05-03T17:20:33.171046+0000","last_active":"2024-05-03T17:21:07.592586+0000","last_peered":"2024-05-03T17:21:07.592586+0000","last_clean":"2024-05-03T17:21:07.592586+0000","last_became_active":"2024-05-03T17:20:33.169410+0000","last_became_peered":"2024-05-03T17:20:33.169410+0000","last_unstale":"2024-05-03T17:21:07.592586+0000","last_undegraded":"2024-05-03T17:21:07.592586+0000","last_fullsized":"2024-05-03T17:21:07.592586+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:45:05.987672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604346+0000","last_change":"2024-05-03T17:20:37.210695+0000","last_active":"2024-05-03T17:21:07.604346+0000","last_peered":"2024-05-03T17:21:07.604346+0000","last_clean":"2024-05-03T17:21:07.604346+0000","last_became_active":"2024-05-03T17:20:37.210285+0000","last_became_peered":"2024-05-03T17:20:37.210285+0000","last_unstale":"2024-05-03T17:21:07.604346+0000","last_undegraded":"2024-05-03T17:21:07.604346+0000","last_fullsized":"2024-05-03T17:21:07.604346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:10:52.013017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595054+0000","last_change":"2024-05-03T17:20:33.163997+0000","last_active":"2024-05-03T17:21:07.595054+0000","last_peered":"2024-05-03T17:21:07.595054+0000","last_clean":"2024-05-03T17:21:07.595054+0000","last_became_active":"2024-05-03T17:20:33.163653+0000","last_became_peered":"2024-05-03T17:20:33.163653+0000","last_unstale":"2024-05-03T17:21:07.595054+0000","last_undegraded":"2024-05-03T17:21:07.595054+0000","last_fullsized":"2024-05-03T17:21:07.595054+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:55:51.762133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592099+0000","last_change":"2024-05-03T17:20:37.195967+0000","last_active":"2024-05-03T17:21:07.592099+0000","last_peered":"2024-05-03T17:21:07.592099+0000","last_clean":"2024-05-03T17:21:07.592099+0000","last_became_active":"2024-05-03T17:20:37.195652+0000","last_became_peered":"2024-05-03T17:20:37.195652+0000","last_unstale":"2024-05-03T17:21:07.592099+0000","last_undegraded":"2024-05-03T17:21:07.592099+0000","last_fullsized":"2024-05-03T17:21:07.592099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:34:37.381525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603936+0000","last_change":"2024-05-03T17:20:33.167378+0000","last_active":"2024-05-03T17:21:07.603936+0000","last_peered":"2024-05-03T17:21:07.603936+0000","last_clean":"2024-05-03T17:21:07.603936+0000","last_became_active":"2024-05-03T17:20:33.167000+0000","last_became_peered":"2024-05-03T17:20:33.167000+0000","last_unstale":"2024-05-03T17:21:07.603936+0000","last_undegraded":"2024-05-03T17:21:07.603936+0000","last_fullsized":"2024-05-03T17:21:07.603936+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:53:48.087190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598616+0000","last_change":"2024-05-03T17:21:07.598616+0000","last_active":"2024-05-03T17:21:07.598616+0000","last_peered":"2024-05-03T17:21:07.598616+0000","last_clean":"2024-05-03T17:21:07.598616+0000","last_became_active":"2024-05-03T17:21:07.598234+0000","last_became_peered":"2024-05-03T17:21:07.598234+0000","last_unstale":"2024-05-03T17:21:07.598616+0000","last_undegraded":"2024-05-03T17:21:07.598616+0000","last_fullsized":"2024-05-03T17:21:07.598616+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:18:59.968590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'3","reported_seq":53,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:20.621594+0000","last_change":"2024-05-03T17:20:33.172079+0000","last_active":"2024-05-03T17:21:20.621594+0000","last_peered":"2024-05-03T17:21:20.621594+0000","last_clean":"2024-05-03T17:21:20.621594+0000","last_became_active":"2024-05-03T17:20:33.171323+0000","last_became_peered":"2024-05-03T17:20:33.171323+0000","last_unstale":"2024-05-03T17:21:20.621594+0000","last_undegraded":"2024-05-03T17:21:20.621594+0000","last_fullsized":"2024-05-03T17:21:20.621594+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:52:16.091897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599057+0000","last_change":"2024-05-03T17:20:37.202340+0000","last_active":"2024-05-03T17:21:07.599057+0000","last_peered":"2024-05-03T17:21:07.599057+0000","last_clean":"2024-05-03T17:21:07.599057+0000","last_became_active":"2024-05-03T17:20:37.201986+0000","last_became_peered":"2024-05-03T17:20:37.201986+0000","last_unstale":"2024-05-03T17:21:07.599057+0000","last_undegraded":"2024-05-03T17:21:07.599057+0000","last_fullsized":"2024-05-03T17:21:07.599057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:13:47.492653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.611413+0000","last_change":"2024-05-03T17:21:07.611413+0000","last_active":"2024-05-03T17:21:07.611413+0000","last_peered":"2024-05-03T17:21:07.611413+0000","last_clean":"2024-05-03T17:21:07.611413+0000","last_became_active":"2024-05-03T17:21:07.604448+0000","last_became_peered":"2024-05-03T17:21:07.604448+0000","last_unstale":"2024-05-03T17:21:07.611413+0000","last_undegraded":"2024-05-03T17:21:07.611413+0000","last_fullsized":"2024-05-03T17:21:07.611413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:28:29.639741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604785+0000","last_change":"2024-05-03T17:21:07.604785+0000","last_active":"2024-05-03T17:21:07.604785+0000","last_peered":"2024-05-03T17:21:07.604785+0000","last_clean":"2024-05-03T17:21:07.604785+0000","last_became_active":"2024-05-03T17:21:07.604507+0000","last_became_peered":"2024-05-03T17:21:07.604507+0000","last_unstale":"2024-05-03T17:21:07.604785+0000","last_undegraded":"2024-05-03T17:21:07.604785+0000","last_fullsized":"2024-05-03T17:21:07.604785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:53:52.501538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604562+0000","last_change":"2024-05-03T17:20:33.284585+0000","last_active":"2024-05-03T17:21:07.604562+0000","last_peered":"2024-05-03T17:21:07.604562+0000","last_clean":"2024-05-03T17:21:07.604562+0000","last_became_active":"2024-05-03T17:20:33.284203+0000","last_became_peered":"2024-05-03T17:20:33.284203+0000","last_unstale":"2024-05-03T17:21:07.604562+0000","last_undegraded":"2024-05-03T17:21:07.604562+0000","last_fullsized":"2024-05-03T17:21:07.604562+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:40:34.608382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598750+0000","last_change":"2024-05-03T17:20:37.203127+0000","last_active":"2024-05-03T17:21:07.598750+0000","last_peered":"2024-05-03T17:21:07.598750+0000","last_clean":"2024-05-03T17:21:07.598750+0000","last_became_active":"2024-05-03T17:20:37.202809+0000","last_became_peered":"2024-05-03T17:20:37.202809+0000","last_unstale":"2024-05-03T17:21:07.598750+0000","last_undegraded":"2024-05-03T17:21:07.598750+0000","last_fullsized":"2024-05-03T17:21:07.598750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:36:38.136107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'25","reported_seq":106,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:21.624737+0000","last_change":"2024-05-03T17:20:33.171487+0000","last_active":"2024-05-03T17:21:21.624737+0000","last_peered":"2024-05-03T17:21:21.624737+0000","last_clean":"2024-05-03T17:21:21.624737+0000","last_became_active":"2024-05-03T17:20:33.171021+0000","last_became_peered":"2024-05-03T17:20:33.171021+0000","last_unstale":"2024-05-03T17:21:21.624737+0000","last_undegraded":"2024-05-03T17:21:21.624737+0000","last_fullsized":"2024-05-03T17:21:21.624737+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:20:05.729189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50695,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":55,"num_read_kb":76,"num_write":26,"num_write_kb":57,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597839+0000","last_change":"2024-05-03T17:20:37.208443+0000","last_active":"2024-05-03T17:21:07.597839+0000","last_peered":"2024-05-03T17:21:07.597839+0000","last_clean":"2024-05-03T17:21:07.597839+0000","last_became_active":"2024-05-03T17:20:37.207869+0000","last_became_peered":"2024-05-03T17:20:37.207869+0000","last_unstale":"2024-05-03T17:21:07.597839+0000","last_undegraded":"2024-05-03T17:21:07.597839+0000","last_fullsized":"2024-05-03T17:21:07.597839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:13:28.978735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594065+0000","last_change":"2024-05-03T17:20:33.167368+0000","last_active":"2024-05-03T17:21:07.594065+0000","last_peered":"2024-05-03T17:21:07.594065+0000","last_clean":"2024-05-03T17:21:07.594065+0000","last_became_active":"2024-05-03T17:20:33.166503+0000","last_became_peered":"2024-05-03T17:20:33.166503+0000","last_unstale":"2024-05-03T17:21:07.594065+0000","last_undegraded":"2024-05-03T17:21:07.594065+0000","last_fullsized":"2024-05-03T17:21:07.594065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:14:23.150421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595589+0000","last_change":"2024-05-03T17:20:37.201887+0000","last_active":"2024-05-03T17:21:07.595589+0000","last_peered":"2024-05-03T17:21:07.595589+0000","last_clean":"2024-05-03T17:21:07.595589+0000","last_became_active":"2024-05-03T17:20:37.201548+0000","last_became_peered":"2024-05-03T17:20:37.201548+0000","last_unstale":"2024-05-03T17:21:07.595589+0000","last_undegraded":"2024-05-03T17:21:07.595589+0000","last_fullsized":"2024-05-03T17:21:07.595589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:39:26.416885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604320+0000","last_change":"2024-05-03T17:20:33.168039+0000","last_active":"2024-05-03T17:21:07.604320+0000","last_peered":"2024-05-03T17:21:07.604320+0000","last_clean":"2024-05-03T17:21:07.604320+0000","last_became_active":"2024-05-03T17:20:33.165695+0000","last_became_peered":"2024-05-03T17:20:33.165695+0000","last_unstale":"2024-05-03T17:21:07.604320+0000","last_undegraded":"2024-05-03T17:21:07.604320+0000","last_fullsized":"2024-05-03T17:21:07.604320+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:47:17.964849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596442+0000","last_change":"2024-05-03T17:20:37.207618+0000","last_active":"2024-05-03T17:21:07.596442+0000","last_peered":"2024-05-03T17:21:07.596442+0000","last_clean":"2024-05-03T17:21:07.596442+0000","last_became_active":"2024-05-03T17:20:37.207290+0000","last_became_peered":"2024-05-03T17:20:37.207290+0000","last_unstale":"2024-05-03T17:21:07.596442+0000","last_undegraded":"2024-05-03T17:21:07.596442+0000","last_fullsized":"2024-05-03T17:21:07.596442+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:26:34.085385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'3","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:21.623169+0000","last_change":"2024-05-03T17:20:33.167328+0000","last_active":"2024-05-03T17:21:21.623169+0000","last_peered":"2024-05-03T17:21:21.623169+0000","last_clean":"2024-05-03T17:21:21.623169+0000","last_became_active":"2024-05-03T17:20:33.166920+0000","last_became_peered":"2024-05-03T17:20:33.166920+0000","last_unstale":"2024-05-03T17:21:21.623169+0000","last_undegraded":"2024-05-03T17:21:21.623169+0000","last_fullsized":"2024-05-03T17:21:21.623169+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T19:04:47.450596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596215+0000","last_change":"2024-05-03T17:20:37.212691+0000","last_active":"2024-05-03T17:21:07.596215+0000","last_peered":"2024-05-03T17:21:07.596215+0000","last_clean":"2024-05-03T17:21:07.596215+0000","last_became_active":"2024-05-03T17:20:37.208692+0000","last_became_peered":"2024-05-03T17:20:37.208692+0000","last_unstale":"2024-05-03T17:21:07.596215+0000","last_undegraded":"2024-05-03T17:21:07.596215+0000","last_fullsized":"2024-05-03T17:21:07.596215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:04:29.441208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597477+0000","last_change":"2024-05-03T17:20:33.285266+0000","last_active":"2024-05-03T17:21:07.597477+0000","last_peered":"2024-05-03T17:21:07.597477+0000","last_clean":"2024-05-03T17:21:07.597477+0000","last_became_active":"2024-05-03T17:20:33.284223+0000","last_became_peered":"2024-05-03T17:20:33.284223+0000","last_unstale":"2024-05-03T17:21:07.597477+0000","last_undegraded":"2024-05-03T17:21:07.597477+0000","last_fullsized":"2024-05-03T17:21:07.597477+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:20:49.077832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604104+0000","last_change":"2024-05-03T17:20:37.205094+0000","last_active":"2024-05-03T17:21:07.604104+0000","last_peered":"2024-05-03T17:21:07.604104+0000","last_clean":"2024-05-03T17:21:07.604104+0000","last_became_active":"2024-05-03T17:20:37.204769+0000","last_became_peered":"2024-05-03T17:20:37.204769+0000","last_unstale":"2024-05-03T17:21:07.604104+0000","last_undegraded":"2024-05-03T17:21:07.604104+0000","last_fullsized":"2024-05-03T17:21:07.604104+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:00:32.129259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598498+0000","last_change":"2024-05-03T17:20:33.167920+0000","last_active":"2024-05-03T17:21:07.598498+0000","last_peered":"2024-05-03T17:21:07.598498+0000","last_clean":"2024-05-03T17:21:07.598498+0000","last_became_active":"2024-05-03T17:20:33.167571+0000","last_became_peered":"2024-05-03T17:20:33.167571+0000","last_unstale":"2024-05-03T17:21:07.598498+0000","last_undegraded":"2024-05-03T17:21:07.598498+0000","last_fullsized":"2024-05-03T17:21:07.598498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:13:23.462127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595627+0000","last_change":"2024-05-03T17:20:37.208051+0000","last_active":"2024-05-03T17:21:07.595627+0000","last_peered":"2024-05-03T17:21:07.595627+0000","last_clean":"2024-05-03T17:21:07.595627+0000","last_became_active":"2024-05-03T17:20:37.207685+0000","last_became_peered":"2024-05-03T17:20:37.207685+0000","last_unstale":"2024-05-03T17:21:07.595627+0000","last_undegraded":"2024-05-03T17:21:07.595627+0000","last_fullsized":"2024-05-03T17:21:07.595627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:27:47.685727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593658+0000","last_change":"2024-05-03T17:20:33.285337+0000","last_active":"2024-05-03T17:21:07.593658+0000","last_peered":"2024-05-03T17:21:07.593658+0000","last_clean":"2024-05-03T17:21:07.593658+0000","last_became_active":"2024-05-03T17:20:33.285012+0000","last_became_peered":"2024-05-03T17:20:33.285012+0000","last_unstale":"2024-05-03T17:21:07.593658+0000","last_undegraded":"2024-05-03T17:21:07.593658+0000","last_fullsized":"2024-05-03T17:21:07.593658+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:04:44.246674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593083+0000","last_change":"2024-05-03T17:20:37.211216+0000","last_active":"2024-05-03T17:21:07.593083+0000","last_peered":"2024-05-03T17:21:07.593083+0000","last_clean":"2024-05-03T17:21:07.593083+0000","last_became_active":"2024-05-03T17:20:37.210906+0000","last_became_peered":"2024-05-03T17:20:37.210906+0000","last_unstale":"2024-05-03T17:21:07.593083+0000","last_undegraded":"2024-05-03T17:21:07.593083+0000","last_fullsized":"2024-05-03T17:21:07.593083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:20:45.831133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597030+0000","last_change":"2024-05-03T17:20:33.165952+0000","last_active":"2024-05-03T17:21:07.597030+0000","last_peered":"2024-05-03T17:21:07.597030+0000","last_clean":"2024-05-03T17:21:07.597030+0000","last_became_active":"2024-05-03T17:20:33.165546+0000","last_became_peered":"2024-05-03T17:20:33.165546+0000","last_unstale":"2024-05-03T17:21:07.597030+0000","last_undegraded":"2024-05-03T17:21:07.597030+0000","last_fullsized":"2024-05-03T17:21:07.597030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:26:44.484883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592776+0000","last_change":"2024-05-03T17:20:37.209107+0000","last_active":"2024-05-03T17:21:07.592776+0000","last_peered":"2024-05-03T17:21:07.592776+0000","last_clean":"2024-05-03T17:21:07.592776+0000","last_became_active":"2024-05-03T17:20:37.208754+0000","last_became_peered":"2024-05-03T17:20:37.208754+0000","last_unstale":"2024-05-03T17:21:07.592776+0000","last_undegraded":"2024-05-03T17:21:07.592776+0000","last_fullsized":"2024-05-03T17:21:07.592776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:47:22.142530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604237+0000","last_change":"2024-05-03T17:20:37.209092+0000","last_active":"2024-05-03T17:21:07.604237+0000","last_peered":"2024-05-03T17:21:07.604237+0000","last_clean":"2024-05-03T17:21:07.604237+0000","last_became_active":"2024-05-03T17:20:37.208618+0000","last_became_peered":"2024-05-03T17:20:37.208618+0000","last_unstale":"2024-05-03T17:21:07.604237+0000","last_undegraded":"2024-05-03T17:21:07.604237+0000","last_fullsized":"2024-05-03T17:21:07.604237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:11:09.509930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'4","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:22.623941+0000","last_change":"2024-05-03T17:20:33.283985+0000","last_active":"2024-05-03T17:21:22.623941+0000","last_peered":"2024-05-03T17:21:22.623941+0000","last_clean":"2024-05-03T17:21:22.623941+0000","last_became_active":"2024-05-03T17:20:33.283629+0000","last_became_peered":"2024-05-03T17:20:33.283629+0000","last_unstale":"2024-05-03T17:21:22.623941+0000","last_undegraded":"2024-05-03T17:21:22.623941+0000","last_fullsized":"2024-05-03T17:21:22.623941+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:03:08.751017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592649+0000","last_change":"2024-05-03T17:20:37.209086+0000","last_active":"2024-05-03T17:21:07.592649+0000","last_peered":"2024-05-03T17:21:07.592649+0000","last_clean":"2024-05-03T17:21:07.592649+0000","last_became_active":"2024-05-03T17:20:37.207968+0000","last_became_peered":"2024-05-03T17:20:37.207968+0000","last_unstale":"2024-05-03T17:21:07.592649+0000","last_undegraded":"2024-05-03T17:21:07.592649+0000","last_fullsized":"2024-05-03T17:21:07.592649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:57:51.816995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.602944+0000","last_change":"2024-05-03T17:20:33.164302+0000","last_active":"2024-05-03T17:21:07.602944+0000","last_peered":"2024-05-03T17:21:07.602944+0000","last_clean":"2024-05-03T17:21:07.602944+0000","last_became_active":"2024-05-03T17:20:33.163958+0000","last_became_peered":"2024-05-03T17:20:33.163958+0000","last_unstale":"2024-05-03T17:21:07.602944+0000","last_undegraded":"2024-05-03T17:21:07.602944+0000","last_fullsized":"2024-05-03T17:21:07.602944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:09:08.221659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":100,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596315+0000","last_change":"2024-05-03T17:19:04.188548+0000","last_active":"2024-05-03T17:21:07.596315+0000","last_peered":"2024-05-03T17:21:07.596315+0000","last_clean":"2024-05-03T17:21:07.596315+0000","last_became_active":"2024-05-03T17:19:04.183986+0000","last_became_peered":"2024-05-03T17:19:04.183986+0000","last_unstale":"2024-05-03T17:21:07.596315+0000","last_undegraded":"2024-05-03T17:21:07.596315+0000","last_fullsized":"2024-05-03T17:21:07.596315+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:15:37.276361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:15:37.276361+0000","last_clean_scrub_stamp":"2024-05-03T17:15:37.276361+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:45:57.109987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594820+0000","last_change":"2024-05-03T17:20:37.206686+0000","last_active":"2024-05-03T17:21:07.594820+0000","last_peered":"2024-05-03T17:21:07.594820+0000","last_clean":"2024-05-03T17:21:07.594820+0000","last_became_active":"2024-05-03T17:20:37.204810+0000","last_became_peered":"2024-05-03T17:20:37.204810+0000","last_unstale":"2024-05-03T17:21:07.594820+0000","last_undegraded":"2024-05-03T17:21:07.594820+0000","last_fullsized":"2024-05-03T17:21:07.594820+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:52:11.363372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596472+0000","last_change":"2024-05-03T17:20:33.168412+0000","last_active":"2024-05-03T17:21:07.596472+0000","last_peered":"2024-05-03T17:21:07.596472+0000","last_clean":"2024-05-03T17:21:07.596472+0000","last_became_active":"2024-05-03T17:20:33.168089+0000","last_became_peered":"2024-05-03T17:20:33.168089+0000","last_unstale":"2024-05-03T17:21:07.596472+0000","last_undegraded":"2024-05-03T17:21:07.596472+0000","last_fullsized":"2024-05-03T17:21:07.596472+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:03:11.983138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.602992+0000","last_change":"2024-05-03T17:21:07.602540+0000","last_active":"2024-05-03T17:21:07.602992+0000","last_peered":"2024-05-03T17:21:07.602992+0000","last_clean":"2024-05-03T17:21:07.602992+0000","last_became_active":"2024-05-03T17:21:07.599329+0000","last_became_peered":"2024-05-03T17:21:07.599329+0000","last_unstale":"2024-05-03T17:21:07.602992+0000","last_undegraded":"2024-05-03T17:21:07.602992+0000","last_fullsized":"2024-05-03T17:21:07.602992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:57:16.367541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594903+0000","last_change":"2024-05-03T17:20:37.206615+0000","last_active":"2024-05-03T17:21:07.594903+0000","last_peered":"2024-05-03T17:21:07.594903+0000","last_clean":"2024-05-03T17:21:07.594903+0000","last_became_active":"2024-05-03T17:20:37.205519+0000","last_became_peered":"2024-05-03T17:20:37.205519+0000","last_unstale":"2024-05-03T17:21:07.594903+0000","last_undegraded":"2024-05-03T17:21:07.594903+0000","last_fullsized":"2024-05-03T17:21:07.594903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T19:08:45.459749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'10","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:18.215255+0000","last_change":"2024-05-03T17:20:33.164933+0000","last_active":"2024-05-03T17:21:18.215255+0000","last_peered":"2024-05-03T17:21:18.215255+0000","last_clean":"2024-05-03T17:21:18.215255+0000","last_became_active":"2024-05-03T17:20:33.164403+0000","last_became_peered":"2024-05-03T17:20:33.164403+0000","last_unstale":"2024-05-03T17:21:18.215255+0000","last_undegraded":"2024-05-03T17:21:18.215255+0000","last_fullsized":"2024-05-03T17:21:18.215255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:11:21.158371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592505+0000","last_change":"2024-05-03T17:20:37.211145+0000","last_active":"2024-05-03T17:21:07.592505+0000","last_peered":"2024-05-03T17:21:07.592505+0000","last_clean":"2024-05-03T17:21:07.592505+0000","last_became_active":"2024-05-03T17:20:37.210829+0000","last_became_peered":"2024-05-03T17:20:37.210829+0000","last_unstale":"2024-05-03T17:21:07.592505+0000","last_undegraded":"2024-05-03T17:21:07.592505+0000","last_fullsized":"2024-05-03T17:21:07.592505+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:19:23.868001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603017+0000","last_change":"2024-05-03T17:20:33.166287+0000","last_active":"2024-05-03T17:21:07.603017+0000","last_peered":"2024-05-03T17:21:07.603017+0000","last_clean":"2024-05-03T17:21:07.603017+0000","last_became_active":"2024-05-03T17:20:33.165929+0000","last_became_peered":"2024-05-03T17:20:33.165929+0000","last_unstale":"2024-05-03T17:21:07.603017+0000","last_undegraded":"2024-05-03T17:21:07.603017+0000","last_fullsized":"2024-05-03T17:21:07.603017+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:32:37.093941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599190+0000","last_change":"2024-05-03T17:20:37.186668+0000","last_active":"2024-05-03T17:21:07.599190+0000","last_peered":"2024-05-03T17:21:07.599190+0000","last_clean":"2024-05-03T17:21:07.599190+0000","last_became_active":"2024-05-03T17:20:37.185922+0000","last_became_peered":"2024-05-03T17:20:37.185922+0000","last_unstale":"2024-05-03T17:21:07.599190+0000","last_undegraded":"2024-05-03T17:21:07.599190+0000","last_fullsized":"2024-05-03T17:21:07.599190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:00:11.075950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598383+0000","last_change":"2024-05-03T17:20:33.284515+0000","last_active":"2024-05-03T17:21:07.598383+0000","last_peered":"2024-05-03T17:21:07.598383+0000","last_clean":"2024-05-03T17:21:07.598383+0000","last_became_active":"2024-05-03T17:20:33.284169+0000","last_became_peered":"2024-05-03T17:20:33.284169+0000","last_unstale":"2024-05-03T17:21:07.598383+0000","last_undegraded":"2024-05-03T17:21:07.598383+0000","last_fullsized":"2024-05-03T17:21:07.598383+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:27:49.722159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597482+0000","last_change":"2024-05-03T17:20:37.200724+0000","last_active":"2024-05-03T17:21:07.597482+0000","last_peered":"2024-05-03T17:21:07.597482+0000","last_clean":"2024-05-03T17:21:07.597482+0000","last_became_active":"2024-05-03T17:20:37.200383+0000","last_became_peered":"2024-05-03T17:20:37.200383+0000","last_unstale":"2024-05-03T17:21:07.597482+0000","last_undegraded":"2024-05-03T17:21:07.597482+0000","last_fullsized":"2024-05-03T17:21:07.597482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:12:15.547433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592389+0000","last_change":"2024-05-03T17:20:33.163642+0000","last_active":"2024-05-03T17:21:07.592389+0000","last_peered":"2024-05-03T17:21:07.592389+0000","last_clean":"2024-05-03T17:21:07.592389+0000","last_became_active":"2024-05-03T17:20:33.163270+0000","last_became_peered":"2024-05-03T17:20:33.163270+0000","last_unstale":"2024-05-03T17:21:07.592389+0000","last_undegraded":"2024-05-03T17:21:07.592389+0000","last_fullsized":"2024-05-03T17:21:07.592389+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:48:02.719294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595341+0000","last_change":"2024-05-03T17:20:37.203063+0000","last_active":"2024-05-03T17:21:07.595341+0000","last_peered":"2024-05-03T17:21:07.595341+0000","last_clean":"2024-05-03T17:21:07.595341+0000","last_became_active":"2024-05-03T17:20:37.202609+0000","last_became_peered":"2024-05-03T17:20:37.202609+0000","last_unstale":"2024-05-03T17:21:07.595341+0000","last_undegraded":"2024-05-03T17:21:07.595341+0000","last_fullsized":"2024-05-03T17:21:07.595341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:38:08.497937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593609+0000","last_change":"2024-05-03T17:20:33.168672+0000","last_active":"2024-05-03T17:21:07.593609+0000","last_peered":"2024-05-03T17:21:07.593609+0000","last_clean":"2024-05-03T17:21:07.593609+0000","last_became_active":"2024-05-03T17:20:33.168349+0000","last_became_peered":"2024-05-03T17:20:33.168349+0000","last_unstale":"2024-05-03T17:21:07.593609+0000","last_undegraded":"2024-05-03T17:21:07.593609+0000","last_fullsized":"2024-05-03T17:21:07.593609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:26:04.340040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592378+0000","last_change":"2024-05-03T17:20:37.203725+0000","last_active":"2024-05-03T17:21:07.592378+0000","last_peered":"2024-05-03T17:21:07.592378+0000","last_clean":"2024-05-03T17:21:07.592378+0000","last_became_active":"2024-05-03T17:20:37.203318+0000","last_became_peered":"2024-05-03T17:20:37.203318+0000","last_unstale":"2024-05-03T17:21:07.592378+0000","last_undegraded":"2024-05-03T17:21:07.592378+0000","last_fullsized":"2024-05-03T17:21:07.592378+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:00:48.706036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.609823+0000","last_change":"2024-05-03T17:21:07.609392+0000","last_active":"2024-05-03T17:21:07.609823+0000","last_peered":"2024-05-03T17:21:07.609823+0000","last_clean":"2024-05-03T17:21:07.609823+0000","last_became_active":"2024-05-03T17:21:07.600420+0000","last_became_peered":"2024-05-03T17:21:07.600420+0000","last_unstale":"2024-05-03T17:21:07.609823+0000","last_undegraded":"2024-05-03T17:21:07.609823+0000","last_fullsized":"2024-05-03T17:21:07.609823+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:03:47.869058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597082+0000","last_change":"2024-05-03T17:20:37.203766+0000","last_active":"2024-05-03T17:21:07.597082+0000","last_peered":"2024-05-03T17:21:07.597082+0000","last_clean":"2024-05-03T17:21:07.597082+0000","last_became_active":"2024-05-03T17:20:37.203306+0000","last_became_peered":"2024-05-03T17:20:37.203306+0000","last_unstale":"2024-05-03T17:21:07.597082+0000","last_undegraded":"2024-05-03T17:21:07.597082+0000","last_fullsized":"2024-05-03T17:21:07.597082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:57:22.760234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594059+0000","last_change":"2024-05-03T17:20:33.285014+0000","last_active":"2024-05-03T17:21:07.594059+0000","last_peered":"2024-05-03T17:21:07.594059+0000","last_clean":"2024-05-03T17:21:07.594059+0000","last_became_active":"2024-05-03T17:20:33.284178+0000","last_became_peered":"2024-05-03T17:20:33.284178+0000","last_unstale":"2024-05-03T17:21:07.594059+0000","last_undegraded":"2024-05-03T17:21:07.594059+0000","last_fullsized":"2024-05-03T17:21:07.594059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:45:57.315989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597013+0000","last_change":"2024-05-03T17:20:37.201286+0000","last_active":"2024-05-03T17:21:07.597013+0000","last_peered":"2024-05-03T17:21:07.597013+0000","last_clean":"2024-05-03T17:21:07.597013+0000","last_became_active":"2024-05-03T17:20:37.200879+0000","last_became_peered":"2024-05-03T17:20:37.200879+0000","last_unstale":"2024-05-03T17:21:07.597013+0000","last_undegraded":"2024-05-03T17:21:07.597013+0000","last_fullsized":"2024-05-03T17:21:07.597013+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:25:49.936431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604390+0000","last_change":"2024-05-03T17:20:33.165850+0000","last_active":"2024-05-03T17:21:07.604390+0000","last_peered":"2024-05-03T17:21:07.604390+0000","last_clean":"2024-05-03T17:21:07.604390+0000","last_became_active":"2024-05-03T17:20:33.165478+0000","last_became_peered":"2024-05-03T17:20:33.165478+0000","last_unstale":"2024-05-03T17:21:07.604390+0000","last_undegraded":"2024-05-03T17:21:07.604390+0000","last_fullsized":"2024-05-03T17:21:07.604390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:22:36.224538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596692+0000","last_change":"2024-05-03T17:20:37.210896+0000","last_active":"2024-05-03T17:21:07.596692+0000","last_peered":"2024-05-03T17:21:07.596692+0000","last_clean":"2024-05-03T17:21:07.596692+0000","last_became_active":"2024-05-03T17:20:37.209150+0000","last_became_peered":"2024-05-03T17:20:37.209150+0000","last_unstale":"2024-05-03T17:21:07.596692+0000","last_undegraded":"2024-05-03T17:21:07.596692+0000","last_fullsized":"2024-05-03T17:21:07.596692+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:17:01.870395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594502+0000","last_change":"2024-05-03T17:20:33.169631+0000","last_active":"2024-05-03T17:21:07.594502+0000","last_peered":"2024-05-03T17:21:07.594502+0000","last_clean":"2024-05-03T17:21:07.594502+0000","last_became_active":"2024-05-03T17:20:33.169293+0000","last_became_peered":"2024-05-03T17:20:33.169293+0000","last_unstale":"2024-05-03T17:21:07.594502+0000","last_undegraded":"2024-05-03T17:21:07.594502+0000","last_fullsized":"2024-05-03T17:21:07.594502+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:20:29.988150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597243+0000","last_change":"2024-05-03T17:20:37.201347+0000","last_active":"2024-05-03T17:21:07.597243+0000","last_peered":"2024-05-03T17:21:07.597243+0000","last_clean":"2024-05-03T17:21:07.597243+0000","last_became_active":"2024-05-03T17:20:37.200999+0000","last_became_peered":"2024-05-03T17:20:37.200999+0000","last_unstale":"2024-05-03T17:21:07.597243+0000","last_undegraded":"2024-05-03T17:21:07.597243+0000","last_fullsized":"2024-05-03T17:21:07.597243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:34:23.027957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595391+0000","last_change":"2024-05-03T17:20:37.206379+0000","last_active":"2024-05-03T17:21:07.595391+0000","last_peered":"2024-05-03T17:21:07.595391+0000","last_clean":"2024-05-03T17:21:07.595391+0000","last_became_active":"2024-05-03T17:20:37.206048+0000","last_became_peered":"2024-05-03T17:20:37.206048+0000","last_unstale":"2024-05-03T17:21:07.595391+0000","last_undegraded":"2024-05-03T17:21:07.595391+0000","last_fullsized":"2024-05-03T17:21:07.595391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:46:02.486170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593768+0000","last_change":"2024-05-03T17:20:37.205997+0000","last_active":"2024-05-03T17:21:07.593768+0000","last_peered":"2024-05-03T17:21:07.593768+0000","last_clean":"2024-05-03T17:21:07.593768+0000","last_became_active":"2024-05-03T17:20:37.205602+0000","last_became_peered":"2024-05-03T17:20:37.205602+0000","last_unstale":"2024-05-03T17:21:07.593768+0000","last_undegraded":"2024-05-03T17:21:07.593768+0000","last_fullsized":"2024-05-03T17:21:07.593768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:16:04.929294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593598+0000","last_change":"2024-05-03T17:20:37.199994+0000","last_active":"2024-05-03T17:21:07.593598+0000","last_peered":"2024-05-03T17:21:07.593598+0000","last_clean":"2024-05-03T17:21:07.593598+0000","last_became_active":"2024-05-03T17:20:37.199653+0000","last_became_peered":"2024-05-03T17:20:37.199653+0000","last_unstale":"2024-05-03T17:21:07.593598+0000","last_undegraded":"2024-05-03T17:21:07.593598+0000","last_fullsized":"2024-05-03T17:21:07.593598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:23:24.945036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598001+0000","last_change":"2024-05-03T17:20:37.198701+0000","last_active":"2024-05-03T17:21:07.598001+0000","last_peered":"2024-05-03T17:21:07.598001+0000","last_clean":"2024-05-03T17:21:07.598001+0000","last_became_active":"2024-05-03T17:20:37.198374+0000","last_became_peered":"2024-05-03T17:20:37.198374+0000","last_unstale":"2024-05-03T17:21:07.598001+0000","last_undegraded":"2024-05-03T17:21:07.598001+0000","last_fullsized":"2024-05-03T17:21:07.598001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:33:26.744746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599746+0000","last_change":"2024-05-03T17:21:07.599746+0000","last_active":"2024-05-03T17:21:07.599746+0000","last_peered":"2024-05-03T17:21:07.599746+0000","last_clean":"2024-05-03T17:21:07.599746+0000","last_became_active":"2024-05-03T17:21:07.598862+0000","last_became_peered":"2024-05-03T17:21:07.598862+0000","last_unstale":"2024-05-03T17:21:07.599746+0000","last_undegraded":"2024-05-03T17:21:07.599746+0000","last_fullsized":"2024-05-03T17:21:07.599746+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:46:31.214548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596253+0000","last_change":"2024-05-03T17:20:37.207194+0000","last_active":"2024-05-03T17:21:07.596253+0000","last_peered":"2024-05-03T17:21:07.596253+0000","last_clean":"2024-05-03T17:21:07.596253+0000","last_became_active":"2024-05-03T17:20:37.206833+0000","last_became_peered":"2024-05-03T17:20:37.206833+0000","last_unstale":"2024-05-03T17:21:07.596253+0000","last_undegraded":"2024-05-03T17:21:07.596253+0000","last_fullsized":"2024-05-03T17:21:07.596253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:29:32.212841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594069+0000","last_change":"2024-05-03T17:20:37.203613+0000","last_active":"2024-05-03T17:21:07.594069+0000","last_peered":"2024-05-03T17:21:07.594069+0000","last_clean":"2024-05-03T17:21:07.594069+0000","last_became_active":"2024-05-03T17:20:37.203265+0000","last_became_peered":"2024-05-03T17:20:37.203265+0000","last_unstale":"2024-05-03T17:21:07.594069+0000","last_undegraded":"2024-05-03T17:21:07.594069+0000","last_fullsized":"2024-05-03T17:21:07.594069+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:12:35.816804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592409+0000","last_change":"2024-05-03T17:20:37.206208+0000","last_active":"2024-05-03T17:21:07.592409+0000","last_peered":"2024-05-03T17:21:07.592409+0000","last_clean":"2024-05-03T17:21:07.592409+0000","last_became_active":"2024-05-03T17:20:37.205859+0000","last_became_peered":"2024-05-03T17:20:37.205859+0000","last_unstale":"2024-05-03T17:21:07.592409+0000","last_undegraded":"2024-05-03T17:21:07.592409+0000","last_fullsized":"2024-05-03T17:21:07.592409+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:57:55.311154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604102+0000","last_change":"2024-05-03T17:20:37.197239+0000","last_active":"2024-05-03T17:21:07.604102+0000","last_peered":"2024-05-03T17:21:07.604102+0000","last_clean":"2024-05-03T17:21:07.604102+0000","last_became_active":"2024-05-03T17:20:37.196903+0000","last_became_peered":"2024-05-03T17:20:37.196903+0000","last_unstale":"2024-05-03T17:21:07.604102+0000","last_undegraded":"2024-05-03T17:21:07.604102+0000","last_fullsized":"2024-05-03T17:21:07.604102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:29:53.020573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592086+0000","last_change":"2024-05-03T17:20:37.211295+0000","last_active":"2024-05-03T17:21:07.592086+0000","last_peered":"2024-05-03T17:21:07.592086+0000","last_clean":"2024-05-03T17:21:07.592086+0000","last_became_active":"2024-05-03T17:20:37.210939+0000","last_became_peered":"2024-05-03T17:20:37.210939+0000","last_unstale":"2024-05-03T17:21:07.592086+0000","last_undegraded":"2024-05-03T17:21:07.592086+0000","last_fullsized":"2024-05-03T17:21:07.592086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:45:58.770190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598703+0000","last_change":"2024-05-03T17:20:37.209581+0000","last_active":"2024-05-03T17:21:07.598703+0000","last_peered":"2024-05-03T17:21:07.598703+0000","last_clean":"2024-05-03T17:21:07.598703+0000","last_became_active":"2024-05-03T17:20:37.209126+0000","last_became_peered":"2024-05-03T17:20:37.209126+0000","last_unstale":"2024-05-03T17:21:07.598703+0000","last_undegraded":"2024-05-03T17:21:07.598703+0000","last_fullsized":"2024-05-03T17:21:07.598703+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:47:15.485725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595388+0000","last_change":"2024-05-03T17:20:37.204258+0000","last_active":"2024-05-03T17:21:07.595388+0000","last_peered":"2024-05-03T17:21:07.595388+0000","last_clean":"2024-05-03T17:21:07.595388+0000","last_became_active":"2024-05-03T17:20:37.202405+0000","last_became_peered":"2024-05-03T17:20:37.202405+0000","last_unstale":"2024-05-03T17:21:07.595388+0000","last_undegraded":"2024-05-03T17:21:07.595388+0000","last_fullsized":"2024-05-03T17:21:07.595388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:36:35.946607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595536+0000","last_change":"2024-05-03T17:20:37.208909+0000","last_active":"2024-05-03T17:21:07.595536+0000","last_peered":"2024-05-03T17:21:07.595536+0000","last_clean":"2024-05-03T17:21:07.595536+0000","last_became_active":"2024-05-03T17:20:37.208540+0000","last_became_peered":"2024-05-03T17:20:37.208540+0000","last_unstale":"2024-05-03T17:21:07.595536+0000","last_undegraded":"2024-05-03T17:21:07.595536+0000","last_fullsized":"2024-05-03T17:21:07.595536+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:32:41.700487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":213988,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":54696,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":170,"num_read_kb":142,"num_write":157,"num_write_kb":98,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":165226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678042,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":876544,"data_stored":428314,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874270,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":1070468,"data_compressed":42001,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037796,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":428528,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234026,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":428426,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430254,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":1020378,"data_compressed":42001,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593780,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32040,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713112,"statfs":{"total":95995035648,"available":95962226688,"internally_reserved":0,"allocated":876544,"data_stored":427917,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":478405,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":897024,"data_stored":429188,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":481284,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":892928,"data_stored":428228,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953534,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":880640,"data_stored":428354,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790009,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":1020062,"data_compressed":42001,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34582,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34582,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":50695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":53574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":52273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-03T17:21:27.390 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-03T17:21:27.639 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:27.905 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:27 smithi067 ceph-mon[29731]: pgmap v321: 97 pgs: 97 active+clean; 630 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-05-03T17:21:27.906 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:27 smithi067 ceph-mon[29731]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:21:28.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:27 smithi082 ceph-mon[33535]: pgmap v321: 97 pgs: 97 active+clean; 630 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-05-03T17:21:28.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:27 smithi082 ceph-mon[33535]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:21:28.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:27 smithi070 ceph-mon[33250]: pgmap v321: 97 pgs: 97 active+clean; 630 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2024-05-03T17:21:28.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:27 smithi070 ceph-mon[33250]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:21:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.358+0000 7fae83855700 1 -- 172.21.15.67:0/3794815681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c1076c0 msgr2=0x7fae7c10f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.358+0000 7fae83855700 1 --2- 172.21.15.67:0/3794815681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c1076c0 0x7fae7c10f290 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fae7400b0d0 tx=0x7fae7400b3e0 comp rx=0 tx=0).stop 2024-05-03T17:21:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.358+0000 7fae83855700 1 -- 172.21.15.67:0/3794815681 shutdown_connections 2024-05-03T17:21:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.358+0000 7fae83855700 1 --2- 172.21.15.67:0/3794815681 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c1076c0 0x7fae7c10f290 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.358+0000 7fae83855700 1 --2- 172.21.15.67:0/3794815681 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae7c102630 0x7fae7c102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.358+0000 7fae83855700 1 --2- 172.21.15.67:0/3794815681 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7c102fe0 0x7fae7c106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.358+0000 7fae83855700 1 -- 172.21.15.67:0/3794815681 >> 172.21.15.67:0/3794815681 conn(0x7fae7c079350 msgr2=0x7fae7c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:29.362 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.358+0000 7fae83855700 1 -- 172.21.15.67:0/3794815681 shutdown_connections 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.358+0000 7fae83855700 1 -- 172.21.15.67:0/3794815681 wait complete. 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.358+0000 7fae83855700 1 Processor -- start 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae83855700 1 -- start start 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae83855700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7c102630 0x7fae7c19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae83855700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c102fe0 0x7fae7c19fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae83855700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae7c1076c0 0x7fae7c199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae83855700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c1125d0 con 0x7fae7c102fe0 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae83855700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c112450 con 0x7fae7c102630 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae83855700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fae7c112750 con 0x7fae7c1076c0 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae80df0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c102fe0 0x7fae7c19fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae80df0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c102fe0 0x7fae7c19fd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42272/0 (socket says 172.21.15.67:42272) 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae80df0700 1 -- 172.21.15.67:0/1554092416 learned_addr learned my addr 172.21.15.67:0/1554092416 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae815f1700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7c102630 0x7fae7c19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:29.363 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae81df2700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae7c1076c0 0x7fae7c199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae80df0700 1 -- 172.21.15.67:0/1554092416 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae7c1076c0 msgr2=0x7fae7c199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae80df0700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae7c1076c0 0x7fae7c199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae80df0700 1 -- 172.21.15.67:0/1554092416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7c102630 msgr2=0x7fae7c19f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae80df0700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7c102630 0x7fae7c19f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.359+0000 7fae80df0700 1 -- 172.21.15.67:0/1554092416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae7c19a160 con 0x7fae7c102fe0 2024-05-03T17:21:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.360+0000 7fae80df0700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c102fe0 0x7fae7c19fd80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fae6c00c9e0 tx=0x7fae6c00ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.360+0000 7fae727fc700 1 -- 172.21.15.67:0/1554092416 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c007aa0 con 0x7fae7c102fe0 2024-05-03T17:21:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.360+0000 7fae727fc700 1 -- 172.21.15.67:0/1554092416 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae6c007c40 con 0x7fae7c102fe0 2024-05-03T17:21:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.360+0000 7fae727fc700 1 -- 172.21.15.67:0/1554092416 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae6c018660 con 0x7fae7c102fe0 2024-05-03T17:21:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.360+0000 7fae83855700 1 -- 172.21.15.67:0/1554092416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae7c19a450 con 0x7fae7c102fe0 2024-05-03T17:21:29.364 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.360+0000 7fae83855700 1 -- 172.21.15.67:0/1554092416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae7c1a6710 con 0x7fae7c102fe0 2024-05-03T17:21:29.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.361+0000 7fae83855700 1 -- 172.21.15.67:0/1554092416 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae7c06a190 con 0x7fae7c102fe0 2024-05-03T17:21:29.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.362+0000 7fae727fc700 1 -- 172.21.15.67:0/1554092416 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fae6c018800 con 0x7fae7c102fe0 2024-05-03T17:21:29.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.363+0000 7fae727fc700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fae680b2910 0x7fae680b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:29.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.363+0000 7fae815f1700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fae680b2910 0x7fae680b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:29.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.363+0000 7fae727fc700 1 -- 172.21.15.67:0/1554092416 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fae6c0e2620 con 0x7fae7c102fe0 2024-05-03T17:21:29.365 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.363+0000 7fae815f1700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fae680b2910 0x7fae680b4dd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fae7c19ab40 tx=0x7fae78009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:29.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.367+0000 7fae727fc700 1 -- 172.21.15.67:0/1554092416 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fae6c014030 con 0x7fae7c102fe0 2024-05-03T17:21:29.580 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.579+0000 7fae83855700 1 -- 172.21.15.67:0/1554092416 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fae7c0638e0 con 0x7fae680b2910 2024-05-03T17:21:29.588 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.586+0000 7fae727fc700 1 -- 172.21.15.67:0/1554092416 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258916 (secure 0 0 0) 0x7fae7c0638e0 con 0x7fae680b2910 2024-05-03T17:21:29.591 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:21:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.589+0000 7fae83855700 1 -- 172.21.15.67:0/1554092416 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fae680b2910 msgr2=0x7fae680b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.589+0000 7fae83855700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fae680b2910 0x7fae680b4dd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fae7c19ab40 tx=0x7fae78009190 comp rx=0 tx=0).stop 2024-05-03T17:21:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.589+0000 7fae83855700 1 -- 172.21.15.67:0/1554092416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c102fe0 msgr2=0x7fae7c19fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:29.593 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.589+0000 7fae83855700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c102fe0 0x7fae7c19fd80 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fae6c00c9e0 tx=0x7fae6c00ccf0 comp rx=0 tx=0).stop 2024-05-03T17:21:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.589+0000 7fae83855700 1 -- 172.21.15.67:0/1554092416 shutdown_connections 2024-05-03T17:21:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.589+0000 7fae83855700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fae680b2910 0x7fae680b4dd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.589+0000 7fae83855700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fae7c1076c0 0x7fae7c199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.589+0000 7fae83855700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fae7c102630 0x7fae7c19f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.589+0000 7fae83855700 1 --2- 172.21.15.67:0/1554092416 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fae7c102fe0 0x7fae7c19fd80 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.589+0000 7fae83855700 1 -- 172.21.15.67:0/1554092416 >> 172.21.15.67:0/1554092416 conn(0x7fae7c079350 msgr2=0x7fae7c06be70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.589+0000 7fae83855700 1 -- 172.21.15.67:0/1554092416 shutdown_connections 2024-05-03T17:21:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:29.590+0000 7fae83855700 1 -- 172.21.15.67:0/1554092416 wait complete. 2024-05-03T17:21:29.594 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-03T17:21:29.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:29 smithi067 ceph-mon[29731]: pgmap v322: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-05-03T17:21:30.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:29 smithi082 ceph-mon[33535]: pgmap v322: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-05-03T17:21:30.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:29 smithi070 ceph-mon[33250]: pgmap v322: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.2 KiB/s wr, 9 op/s 2024-05-03T17:21:30.298 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-05-03T17:21:28.429044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":646436,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":272,"num_read_kb":218,"num_write":284,"num_write_kb":2253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":359904,"kb_used_data":10728,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581920,"statfs":{"total":1151940427776,"available":1151571886080,"internally_reserved":0,"allocated":10985472,"data_stored":7071828,"data_compressed":178158,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":29620,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":89,"num_read_kb":71,"num_write":22,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003032"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595780+0000","last_change":"2024-05-03T17:20:37.204469+0000","last_active":"2024-05-03T17:21:07.595780+0000","last_peered":"2024-05-03T17:21:07.595780+0000","last_clean":"2024-05-03T17:21:07.595780+0000","last_became_active":"2024-05-03T17:20:37.204098+0000","last_became_peered":"2024-05-03T17:20:37.204098+0000","last_unstale":"2024-05-03T17:21:07.595780+0000","last_undegraded":"2024-05-03T17:21:07.595780+0000","last_fullsized":"2024-05-03T17:21:07.595780+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:03:29.875319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599098+0000","last_change":"2024-05-03T17:20:37.201171+0000","last_active":"2024-05-03T17:21:07.599098+0000","last_peered":"2024-05-03T17:21:07.599098+0000","last_clean":"2024-05-03T17:21:07.599098+0000","last_became_active":"2024-05-03T17:20:37.200852+0000","last_became_peered":"2024-05-03T17:20:37.200852+0000","last_unstale":"2024-05-03T17:21:07.599098+0000","last_undegraded":"2024-05-03T17:21:07.599098+0000","last_fullsized":"2024-05-03T17:21:07.599098+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:12:06.549919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598922+0000","last_change":"2024-05-03T17:20:37.201925+0000","last_active":"2024-05-03T17:21:07.598922+0000","last_peered":"2024-05-03T17:21:07.598922+0000","last_clean":"2024-05-03T17:21:07.598922+0000","last_became_active":"2024-05-03T17:20:37.201619+0000","last_became_peered":"2024-05-03T17:20:37.201619+0000","last_unstale":"2024-05-03T17:21:07.598922+0000","last_undegraded":"2024-05-03T17:21:07.598922+0000","last_fullsized":"2024-05-03T17:21:07.598922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:50:19.019887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603123+0000","last_change":"2024-05-03T17:20:37.203535+0000","last_active":"2024-05-03T17:21:07.603123+0000","last_peered":"2024-05-03T17:21:07.603123+0000","last_clean":"2024-05-03T17:21:07.603123+0000","last_became_active":"2024-05-03T17:20:37.203175+0000","last_became_peered":"2024-05-03T17:20:37.203175+0000","last_unstale":"2024-05-03T17:21:07.603123+0000","last_undegraded":"2024-05-03T17:21:07.603123+0000","last_fullsized":"2024-05-03T17:21:07.603123+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:59:41.026709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597618+0000","last_change":"2024-05-03T17:20:37.200283+0000","last_active":"2024-05-03T17:21:07.597618+0000","last_peered":"2024-05-03T17:21:07.597618+0000","last_clean":"2024-05-03T17:21:07.597618+0000","last_became_active":"2024-05-03T17:20:37.199105+0000","last_became_peered":"2024-05-03T17:20:37.199105+0000","last_unstale":"2024-05-03T17:21:07.597618+0000","last_undegraded":"2024-05-03T17:21:07.597618+0000","last_fullsized":"2024-05-03T17:21:07.597618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:56:39.990497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597382+0000","last_change":"2024-05-03T17:21:07.597382+0000","last_active":"2024-05-03T17:21:07.597382+0000","last_peered":"2024-05-03T17:21:07.597382+0000","last_clean":"2024-05-03T17:21:07.597382+0000","last_became_active":"2024-05-03T17:21:07.596935+0000","last_became_peered":"2024-05-03T17:21:07.596935+0000","last_unstale":"2024-05-03T17:21:07.597382+0000","last_undegraded":"2024-05-03T17:21:07.597382+0000","last_fullsized":"2024-05-03T17:21:07.597382+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T19:30:50.200783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596543+0000","last_change":"2024-05-03T17:20:37.211378+0000","last_active":"2024-05-03T17:21:07.596543+0000","last_peered":"2024-05-03T17:21:07.596543+0000","last_clean":"2024-05-03T17:21:07.596543+0000","last_became_active":"2024-05-03T17:20:37.210684+0000","last_became_peered":"2024-05-03T17:20:37.210684+0000","last_unstale":"2024-05-03T17:21:07.596543+0000","last_undegraded":"2024-05-03T17:21:07.596543+0000","last_fullsized":"2024-05-03T17:21:07.596543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:35:24.737186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592032+0000","last_change":"2024-05-03T17:20:37.202989+0000","last_active":"2024-05-03T17:21:07.592032+0000","last_peered":"2024-05-03T17:21:07.592032+0000","last_clean":"2024-05-03T17:21:07.592032+0000","last_became_active":"2024-05-03T17:20:37.199975+0000","last_became_peered":"2024-05-03T17:20:37.199975+0000","last_unstale":"2024-05-03T17:21:07.592032+0000","last_undegraded":"2024-05-03T17:21:07.592032+0000","last_fullsized":"2024-05-03T17:21:07.592032+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:13:43.875472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598772+0000","last_change":"2024-05-03T17:20:37.206754+0000","last_active":"2024-05-03T17:21:07.598772+0000","last_peered":"2024-05-03T17:21:07.598772+0000","last_clean":"2024-05-03T17:21:07.598772+0000","last_became_active":"2024-05-03T17:20:37.206423+0000","last_became_peered":"2024-05-03T17:20:37.206423+0000","last_unstale":"2024-05-03T17:21:07.598772+0000","last_undegraded":"2024-05-03T17:21:07.598772+0000","last_fullsized":"2024-05-03T17:21:07.598772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:10:30.227969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593231+0000","last_change":"2024-05-03T17:20:37.211868+0000","last_active":"2024-05-03T17:21:07.593231+0000","last_peered":"2024-05-03T17:21:07.593231+0000","last_clean":"2024-05-03T17:21:07.593231+0000","last_became_active":"2024-05-03T17:20:37.211486+0000","last_became_peered":"2024-05-03T17:20:37.211486+0000","last_unstale":"2024-05-03T17:21:07.593231+0000","last_undegraded":"2024-05-03T17:21:07.593231+0000","last_fullsized":"2024-05-03T17:21:07.593231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:41:03.966446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596847+0000","last_change":"2024-05-03T17:20:37.206511+0000","last_active":"2024-05-03T17:21:07.596847+0000","last_peered":"2024-05-03T17:21:07.596847+0000","last_clean":"2024-05-03T17:21:07.596847+0000","last_became_active":"2024-05-03T17:20:37.206180+0000","last_became_peered":"2024-05-03T17:20:37.206180+0000","last_unstale":"2024-05-03T17:21:07.596847+0000","last_undegraded":"2024-05-03T17:21:07.596847+0000","last_fullsized":"2024-05-03T17:21:07.596847+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:59:07.277623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603068+0000","last_change":"2024-05-03T17:20:37.204842+0000","last_active":"2024-05-03T17:21:07.603068+0000","last_peered":"2024-05-03T17:21:07.603068+0000","last_clean":"2024-05-03T17:21:07.603068+0000","last_became_active":"2024-05-03T17:20:37.204275+0000","last_became_peered":"2024-05-03T17:20:37.204275+0000","last_unstale":"2024-05-03T17:21:07.603068+0000","last_undegraded":"2024-05-03T17:21:07.603068+0000","last_fullsized":"2024-05-03T17:21:07.603068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:27:00.021413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597215+0000","last_change":"2024-05-03T17:21:07.597215+0000","last_active":"2024-05-03T17:21:07.597215+0000","last_peered":"2024-05-03T17:21:07.597215+0000","last_clean":"2024-05-03T17:21:07.597215+0000","last_became_active":"2024-05-03T17:21:07.596833+0000","last_became_peered":"2024-05-03T17:21:07.596833+0000","last_unstale":"2024-05-03T17:21:07.597215+0000","last_undegraded":"2024-05-03T17:21:07.597215+0000","last_fullsized":"2024-05-03T17:21:07.597215+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:09:41.399761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597704+0000","last_change":"2024-05-03T17:20:37.200759+0000","last_active":"2024-05-03T17:21:07.597704+0000","last_peered":"2024-05-03T17:21:07.597704+0000","last_clean":"2024-05-03T17:21:07.597704+0000","last_became_active":"2024-05-03T17:20:37.200410+0000","last_became_peered":"2024-05-03T17:20:37.200410+0000","last_unstale":"2024-05-03T17:21:07.597704+0000","last_undegraded":"2024-05-03T17:21:07.597704+0000","last_fullsized":"2024-05-03T17:21:07.597704+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:51:03.719792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597446+0000","last_change":"2024-05-03T17:20:37.204534+0000","last_active":"2024-05-03T17:21:07.597446+0000","last_peered":"2024-05-03T17:21:07.597446+0000","last_clean":"2024-05-03T17:21:07.597446+0000","last_became_active":"2024-05-03T17:20:37.204167+0000","last_became_peered":"2024-05-03T17:20:37.204167+0000","last_unstale":"2024-05-03T17:21:07.597446+0000","last_undegraded":"2024-05-03T17:21:07.597446+0000","last_fullsized":"2024-05-03T17:21:07.597446+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:31:35.730465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598463+0000","last_change":"2024-05-03T17:20:37.207658+0000","last_active":"2024-05-03T17:21:07.598463+0000","last_peered":"2024-05-03T17:21:07.598463+0000","last_clean":"2024-05-03T17:21:07.598463+0000","last_became_active":"2024-05-03T17:20:37.207313+0000","last_became_peered":"2024-05-03T17:20:37.207313+0000","last_unstale":"2024-05-03T17:21:07.598463+0000","last_undegraded":"2024-05-03T17:21:07.598463+0000","last_fullsized":"2024-05-03T17:21:07.598463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:47:21.287559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595876+0000","last_change":"2024-05-03T17:20:37.205021+0000","last_active":"2024-05-03T17:21:07.595876+0000","last_peered":"2024-05-03T17:21:07.595876+0000","last_clean":"2024-05-03T17:21:07.595876+0000","last_became_active":"2024-05-03T17:20:37.204630+0000","last_became_peered":"2024-05-03T17:20:37.204630+0000","last_unstale":"2024-05-03T17:21:07.595876+0000","last_undegraded":"2024-05-03T17:21:07.595876+0000","last_fullsized":"2024-05-03T17:21:07.595876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:03:00.318648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593860+0000","last_change":"2024-05-03T17:20:37.201829+0000","last_active":"2024-05-03T17:21:07.593860+0000","last_peered":"2024-05-03T17:21:07.593860+0000","last_clean":"2024-05-03T17:21:07.593860+0000","last_became_active":"2024-05-03T17:20:37.201465+0000","last_became_peered":"2024-05-03T17:20:37.201465+0000","last_unstale":"2024-05-03T17:21:07.593860+0000","last_undegraded":"2024-05-03T17:21:07.593860+0000","last_fullsized":"2024-05-03T17:21:07.593860+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:05:46.844150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594023+0000","last_change":"2024-05-03T17:20:37.199138+0000","last_active":"2024-05-03T17:21:07.594023+0000","last_peered":"2024-05-03T17:21:07.594023+0000","last_clean":"2024-05-03T17:21:07.594023+0000","last_became_active":"2024-05-03T17:20:37.198722+0000","last_became_peered":"2024-05-03T17:20:37.198722+0000","last_unstale":"2024-05-03T17:21:07.594023+0000","last_undegraded":"2024-05-03T17:21:07.594023+0000","last_fullsized":"2024-05-03T17:21:07.594023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:12:04.764356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593787+0000","last_change":"2024-05-03T17:20:33.168111+0000","last_active":"2024-05-03T17:21:07.593787+0000","last_peered":"2024-05-03T17:21:07.593787+0000","last_clean":"2024-05-03T17:21:07.593787+0000","last_became_active":"2024-05-03T17:20:33.167892+0000","last_became_peered":"2024-05-03T17:20:33.167892+0000","last_unstale":"2024-05-03T17:21:07.593787+0000","last_undegraded":"2024-05-03T17:21:07.593787+0000","last_fullsized":"2024-05-03T17:21:07.593787+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:55:45.235287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592219+0000","last_change":"2024-05-03T17:20:37.208673+0000","last_active":"2024-05-03T17:21:07.592219+0000","last_peered":"2024-05-03T17:21:07.592219+0000","last_clean":"2024-05-03T17:21:07.592219+0000","last_became_active":"2024-05-03T17:20:37.208360+0000","last_became_peered":"2024-05-03T17:20:37.208360+0000","last_unstale":"2024-05-03T17:21:07.592219+0000","last_undegraded":"2024-05-03T17:21:07.592219+0000","last_fullsized":"2024-05-03T17:21:07.592219+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:42:02.966225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592593+0000","last_change":"2024-05-03T17:20:33.284052+0000","last_active":"2024-05-03T17:21:07.592593+0000","last_peered":"2024-05-03T17:21:07.592593+0000","last_clean":"2024-05-03T17:21:07.592593+0000","last_became_active":"2024-05-03T17:20:33.283635+0000","last_became_peered":"2024-05-03T17:20:33.283635+0000","last_unstale":"2024-05-03T17:21:07.592593+0000","last_undegraded":"2024-05-03T17:21:07.592593+0000","last_fullsized":"2024-05-03T17:21:07.592593+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:17:51.219359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598804+0000","last_change":"2024-05-03T17:20:37.197321+0000","last_active":"2024-05-03T17:21:07.598804+0000","last_peered":"2024-05-03T17:21:07.598804+0000","last_clean":"2024-05-03T17:21:07.598804+0000","last_became_active":"2024-05-03T17:20:37.196569+0000","last_became_peered":"2024-05-03T17:20:37.196569+0000","last_unstale":"2024-05-03T17:21:07.598804+0000","last_undegraded":"2024-05-03T17:21:07.598804+0000","last_fullsized":"2024-05-03T17:21:07.598804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:38:03.274058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598373+0000","last_change":"2024-05-03T17:20:33.168828+0000","last_active":"2024-05-03T17:21:07.598373+0000","last_peered":"2024-05-03T17:21:07.598373+0000","last_clean":"2024-05-03T17:21:07.598373+0000","last_became_active":"2024-05-03T17:20:33.168408+0000","last_became_peered":"2024-05-03T17:20:33.168408+0000","last_unstale":"2024-05-03T17:21:07.598373+0000","last_undegraded":"2024-05-03T17:21:07.598373+0000","last_fullsized":"2024-05-03T17:21:07.598373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:50:57.790353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598587+0000","last_change":"2024-05-03T17:20:37.199853+0000","last_active":"2024-05-03T17:21:07.598587+0000","last_peered":"2024-05-03T17:21:07.598587+0000","last_clean":"2024-05-03T17:21:07.598587+0000","last_became_active":"2024-05-03T17:20:37.199526+0000","last_became_peered":"2024-05-03T17:20:37.199526+0000","last_unstale":"2024-05-03T17:21:07.598587+0000","last_undegraded":"2024-05-03T17:21:07.598587+0000","last_fullsized":"2024-05-03T17:21:07.598587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:42:45.932795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597857+0000","last_change":"2024-05-03T17:20:33.168050+0000","last_active":"2024-05-03T17:21:07.597857+0000","last_peered":"2024-05-03T17:21:07.597857+0000","last_clean":"2024-05-03T17:21:07.597857+0000","last_became_active":"2024-05-03T17:20:33.167479+0000","last_became_peered":"2024-05-03T17:20:33.167479+0000","last_unstale":"2024-05-03T17:21:07.597857+0000","last_undegraded":"2024-05-03T17:21:07.597857+0000","last_fullsized":"2024-05-03T17:21:07.597857+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:39:32.626940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604399+0000","last_change":"2024-05-03T17:21:07.603996+0000","last_active":"2024-05-03T17:21:07.604399+0000","last_peered":"2024-05-03T17:21:07.604399+0000","last_clean":"2024-05-03T17:21:07.604399+0000","last_became_active":"2024-05-03T17:21:07.603647+0000","last_became_peered":"2024-05-03T17:21:07.603647+0000","last_unstale":"2024-05-03T17:21:07.604399+0000","last_undegraded":"2024-05-03T17:21:07.604399+0000","last_fullsized":"2024-05-03T17:21:07.604399+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:31:07.956642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"79'6","reported_seq":64,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:25.629123+0000","last_change":"2024-05-03T17:20:33.166560+0000","last_active":"2024-05-03T17:21:25.629123+0000","last_peered":"2024-05-03T17:21:25.629123+0000","last_clean":"2024-05-03T17:21:25.629123+0000","last_became_active":"2024-05-03T17:20:33.166136+0000","last_became_peered":"2024-05-03T17:20:33.166136+0000","last_unstale":"2024-05-03T17:21:25.629123+0000","last_undegraded":"2024-05-03T17:21:25.629123+0000","last_fullsized":"2024-05-03T17:21:25.629123+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:38:54.132364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":32,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597525+0000","last_change":"2024-05-03T17:20:37.204336+0000","last_active":"2024-05-03T17:21:07.597525+0000","last_peered":"2024-05-03T17:21:07.597525+0000","last_clean":"2024-05-03T17:21:07.597525+0000","last_became_active":"2024-05-03T17:20:37.203962+0000","last_became_peered":"2024-05-03T17:20:37.203962+0000","last_unstale":"2024-05-03T17:21:07.597525+0000","last_undegraded":"2024-05-03T17:21:07.597525+0000","last_fullsized":"2024-05-03T17:21:07.597525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:29:48.081362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592586+0000","last_change":"2024-05-03T17:20:33.171046+0000","last_active":"2024-05-03T17:21:07.592586+0000","last_peered":"2024-05-03T17:21:07.592586+0000","last_clean":"2024-05-03T17:21:07.592586+0000","last_became_active":"2024-05-03T17:20:33.169410+0000","last_became_peered":"2024-05-03T17:20:33.169410+0000","last_unstale":"2024-05-03T17:21:07.592586+0000","last_undegraded":"2024-05-03T17:21:07.592586+0000","last_fullsized":"2024-05-03T17:21:07.592586+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:45:05.987672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604346+0000","last_change":"2024-05-03T17:20:37.210695+0000","last_active":"2024-05-03T17:21:07.604346+0000","last_peered":"2024-05-03T17:21:07.604346+0000","last_clean":"2024-05-03T17:21:07.604346+0000","last_became_active":"2024-05-03T17:20:37.210285+0000","last_became_peered":"2024-05-03T17:20:37.210285+0000","last_unstale":"2024-05-03T17:21:07.604346+0000","last_undegraded":"2024-05-03T17:21:07.604346+0000","last_fullsized":"2024-05-03T17:21:07.604346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:10:52.013017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595054+0000","last_change":"2024-05-03T17:20:33.163997+0000","last_active":"2024-05-03T17:21:07.595054+0000","last_peered":"2024-05-03T17:21:07.595054+0000","last_clean":"2024-05-03T17:21:07.595054+0000","last_became_active":"2024-05-03T17:20:33.163653+0000","last_became_peered":"2024-05-03T17:20:33.163653+0000","last_unstale":"2024-05-03T17:21:07.595054+0000","last_undegraded":"2024-05-03T17:21:07.595054+0000","last_fullsized":"2024-05-03T17:21:07.595054+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:55:51.762133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592099+0000","last_change":"2024-05-03T17:20:37.195967+0000","last_active":"2024-05-03T17:21:07.592099+0000","last_peered":"2024-05-03T17:21:07.592099+0000","last_clean":"2024-05-03T17:21:07.592099+0000","last_became_active":"2024-05-03T17:20:37.195652+0000","last_became_peered":"2024-05-03T17:20:37.195652+0000","last_unstale":"2024-05-03T17:21:07.592099+0000","last_undegraded":"2024-05-03T17:21:07.592099+0000","last_fullsized":"2024-05-03T17:21:07.592099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:34:37.381525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603936+0000","last_change":"2024-05-03T17:20:33.167378+0000","last_active":"2024-05-03T17:21:07.603936+0000","last_peered":"2024-05-03T17:21:07.603936+0000","last_clean":"2024-05-03T17:21:07.603936+0000","last_became_active":"2024-05-03T17:20:33.167000+0000","last_became_peered":"2024-05-03T17:20:33.167000+0000","last_unstale":"2024-05-03T17:21:07.603936+0000","last_undegraded":"2024-05-03T17:21:07.603936+0000","last_fullsized":"2024-05-03T17:21:07.603936+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:53:48.087190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598616+0000","last_change":"2024-05-03T17:21:07.598616+0000","last_active":"2024-05-03T17:21:07.598616+0000","last_peered":"2024-05-03T17:21:07.598616+0000","last_clean":"2024-05-03T17:21:07.598616+0000","last_became_active":"2024-05-03T17:21:07.598234+0000","last_became_peered":"2024-05-03T17:21:07.598234+0000","last_unstale":"2024-05-03T17:21:07.598616+0000","last_undegraded":"2024-05-03T17:21:07.598616+0000","last_fullsized":"2024-05-03T17:21:07.598616+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:18:59.968590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'3","reported_seq":58,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:25.628376+0000","last_change":"2024-05-03T17:20:33.172079+0000","last_active":"2024-05-03T17:21:25.628376+0000","last_peered":"2024-05-03T17:21:25.628376+0000","last_clean":"2024-05-03T17:21:25.628376+0000","last_became_active":"2024-05-03T17:20:33.171323+0000","last_became_peered":"2024-05-03T17:20:33.171323+0000","last_unstale":"2024-05-03T17:21:25.628376+0000","last_undegraded":"2024-05-03T17:21:25.628376+0000","last_fullsized":"2024-05-03T17:21:25.628376+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:52:16.091897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":29,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599057+0000","last_change":"2024-05-03T17:20:37.202340+0000","last_active":"2024-05-03T17:21:07.599057+0000","last_peered":"2024-05-03T17:21:07.599057+0000","last_clean":"2024-05-03T17:21:07.599057+0000","last_became_active":"2024-05-03T17:20:37.201986+0000","last_became_peered":"2024-05-03T17:20:37.201986+0000","last_unstale":"2024-05-03T17:21:07.599057+0000","last_undegraded":"2024-05-03T17:21:07.599057+0000","last_fullsized":"2024-05-03T17:21:07.599057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:13:47.492653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.611413+0000","last_change":"2024-05-03T17:21:07.611413+0000","last_active":"2024-05-03T17:21:07.611413+0000","last_peered":"2024-05-03T17:21:07.611413+0000","last_clean":"2024-05-03T17:21:07.611413+0000","last_became_active":"2024-05-03T17:21:07.604448+0000","last_became_peered":"2024-05-03T17:21:07.604448+0000","last_unstale":"2024-05-03T17:21:07.611413+0000","last_undegraded":"2024-05-03T17:21:07.611413+0000","last_fullsized":"2024-05-03T17:21:07.611413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:28:29.639741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604785+0000","last_change":"2024-05-03T17:21:07.604785+0000","last_active":"2024-05-03T17:21:07.604785+0000","last_peered":"2024-05-03T17:21:07.604785+0000","last_clean":"2024-05-03T17:21:07.604785+0000","last_became_active":"2024-05-03T17:21:07.604507+0000","last_became_peered":"2024-05-03T17:21:07.604507+0000","last_unstale":"2024-05-03T17:21:07.604785+0000","last_undegraded":"2024-05-03T17:21:07.604785+0000","last_fullsized":"2024-05-03T17:21:07.604785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:53:52.501538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604562+0000","last_change":"2024-05-03T17:20:33.284585+0000","last_active":"2024-05-03T17:21:07.604562+0000","last_peered":"2024-05-03T17:21:07.604562+0000","last_clean":"2024-05-03T17:21:07.604562+0000","last_became_active":"2024-05-03T17:20:33.284203+0000","last_became_peered":"2024-05-03T17:20:33.284203+0000","last_unstale":"2024-05-03T17:21:07.604562+0000","last_undegraded":"2024-05-03T17:21:07.604562+0000","last_fullsized":"2024-05-03T17:21:07.604562+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:40:34.608382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598750+0000","last_change":"2024-05-03T17:20:37.203127+0000","last_active":"2024-05-03T17:21:07.598750+0000","last_peered":"2024-05-03T17:21:07.598750+0000","last_clean":"2024-05-03T17:21:07.598750+0000","last_became_active":"2024-05-03T17:20:37.202809+0000","last_became_peered":"2024-05-03T17:20:37.202809+0000","last_unstale":"2024-05-03T17:21:07.598750+0000","last_undegraded":"2024-05-03T17:21:07.598750+0000","last_fullsized":"2024-05-03T17:21:07.598750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:36:38.136107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'27","reported_seq":119,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:26.630655+0000","last_change":"2024-05-03T17:20:33.171487+0000","last_active":"2024-05-03T17:21:26.630655+0000","last_peered":"2024-05-03T17:21:26.630655+0000","last_clean":"2024-05-03T17:21:26.630655+0000","last_became_active":"2024-05-03T17:20:33.171021+0000","last_became_peered":"2024-05-03T17:20:33.171021+0000","last_unstale":"2024-05-03T17:21:26.630655+0000","last_undegraded":"2024-05-03T17:21:26.630655+0000","last_fullsized":"2024-05-03T17:21:26.630655+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:20:05.729189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51833,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":66,"num_read_kb":83,"num_write":28,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597839+0000","last_change":"2024-05-03T17:20:37.208443+0000","last_active":"2024-05-03T17:21:07.597839+0000","last_peered":"2024-05-03T17:21:07.597839+0000","last_clean":"2024-05-03T17:21:07.597839+0000","last_became_active":"2024-05-03T17:20:37.207869+0000","last_became_peered":"2024-05-03T17:20:37.207869+0000","last_unstale":"2024-05-03T17:21:07.597839+0000","last_undegraded":"2024-05-03T17:21:07.597839+0000","last_fullsized":"2024-05-03T17:21:07.597839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:13:28.978735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594065+0000","last_change":"2024-05-03T17:20:33.167368+0000","last_active":"2024-05-03T17:21:07.594065+0000","last_peered":"2024-05-03T17:21:07.594065+0000","last_clean":"2024-05-03T17:21:07.594065+0000","last_became_active":"2024-05-03T17:20:33.166503+0000","last_became_peered":"2024-05-03T17:20:33.166503+0000","last_unstale":"2024-05-03T17:21:07.594065+0000","last_undegraded":"2024-05-03T17:21:07.594065+0000","last_fullsized":"2024-05-03T17:21:07.594065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:14:23.150421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595589+0000","last_change":"2024-05-03T17:20:37.201887+0000","last_active":"2024-05-03T17:21:07.595589+0000","last_peered":"2024-05-03T17:21:07.595589+0000","last_clean":"2024-05-03T17:21:07.595589+0000","last_became_active":"2024-05-03T17:20:37.201548+0000","last_became_peered":"2024-05-03T17:20:37.201548+0000","last_unstale":"2024-05-03T17:21:07.595589+0000","last_undegraded":"2024-05-03T17:21:07.595589+0000","last_fullsized":"2024-05-03T17:21:07.595589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:39:26.416885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604320+0000","last_change":"2024-05-03T17:20:33.168039+0000","last_active":"2024-05-03T17:21:07.604320+0000","last_peered":"2024-05-03T17:21:07.604320+0000","last_clean":"2024-05-03T17:21:07.604320+0000","last_became_active":"2024-05-03T17:20:33.165695+0000","last_became_peered":"2024-05-03T17:20:33.165695+0000","last_unstale":"2024-05-03T17:21:07.604320+0000","last_undegraded":"2024-05-03T17:21:07.604320+0000","last_fullsized":"2024-05-03T17:21:07.604320+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:47:17.964849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596442+0000","last_change":"2024-05-03T17:20:37.207618+0000","last_active":"2024-05-03T17:21:07.596442+0000","last_peered":"2024-05-03T17:21:07.596442+0000","last_clean":"2024-05-03T17:21:07.596442+0000","last_became_active":"2024-05-03T17:20:37.207290+0000","last_became_peered":"2024-05-03T17:20:37.207290+0000","last_unstale":"2024-05-03T17:21:07.596442+0000","last_undegraded":"2024-05-03T17:21:07.596442+0000","last_fullsized":"2024-05-03T17:21:07.596442+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:26:34.085385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'3","reported_seq":61,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:26.630396+0000","last_change":"2024-05-03T17:20:33.167328+0000","last_active":"2024-05-03T17:21:26.630396+0000","last_peered":"2024-05-03T17:21:26.630396+0000","last_clean":"2024-05-03T17:21:26.630396+0000","last_became_active":"2024-05-03T17:20:33.166920+0000","last_became_peered":"2024-05-03T17:20:33.166920+0000","last_unstale":"2024-05-03T17:21:26.630396+0000","last_undegraded":"2024-05-03T17:21:26.630396+0000","last_fullsized":"2024-05-03T17:21:26.630396+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T19:04:47.450596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596215+0000","last_change":"2024-05-03T17:20:37.212691+0000","last_active":"2024-05-03T17:21:07.596215+0000","last_peered":"2024-05-03T17:21:07.596215+0000","last_clean":"2024-05-03T17:21:07.596215+0000","last_became_active":"2024-05-03T17:20:37.208692+0000","last_became_peered":"2024-05-03T17:20:37.208692+0000","last_unstale":"2024-05-03T17:21:07.596215+0000","last_undegraded":"2024-05-03T17:21:07.596215+0000","last_fullsized":"2024-05-03T17:21:07.596215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:04:29.441208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597477+0000","last_change":"2024-05-03T17:20:33.285266+0000","last_active":"2024-05-03T17:21:07.597477+0000","last_peered":"2024-05-03T17:21:07.597477+0000","last_clean":"2024-05-03T17:21:07.597477+0000","last_became_active":"2024-05-03T17:20:33.284223+0000","last_became_peered":"2024-05-03T17:20:33.284223+0000","last_unstale":"2024-05-03T17:21:07.597477+0000","last_undegraded":"2024-05-03T17:21:07.597477+0000","last_fullsized":"2024-05-03T17:21:07.597477+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:20:49.077832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604104+0000","last_change":"2024-05-03T17:20:37.205094+0000","last_active":"2024-05-03T17:21:07.604104+0000","last_peered":"2024-05-03T17:21:07.604104+0000","last_clean":"2024-05-03T17:21:07.604104+0000","last_became_active":"2024-05-03T17:20:37.204769+0000","last_became_peered":"2024-05-03T17:20:37.204769+0000","last_unstale":"2024-05-03T17:21:07.604104+0000","last_undegraded":"2024-05-03T17:21:07.604104+0000","last_fullsized":"2024-05-03T17:21:07.604104+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:00:32.129259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598498+0000","last_change":"2024-05-03T17:20:33.167920+0000","last_active":"2024-05-03T17:21:07.598498+0000","last_peered":"2024-05-03T17:21:07.598498+0000","last_clean":"2024-05-03T17:21:07.598498+0000","last_became_active":"2024-05-03T17:20:33.167571+0000","last_became_peered":"2024-05-03T17:20:33.167571+0000","last_unstale":"2024-05-03T17:21:07.598498+0000","last_undegraded":"2024-05-03T17:21:07.598498+0000","last_fullsized":"2024-05-03T17:21:07.598498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:13:23.462127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595627+0000","last_change":"2024-05-03T17:20:37.208051+0000","last_active":"2024-05-03T17:21:07.595627+0000","last_peered":"2024-05-03T17:21:07.595627+0000","last_clean":"2024-05-03T17:21:07.595627+0000","last_became_active":"2024-05-03T17:20:37.207685+0000","last_became_peered":"2024-05-03T17:20:37.207685+0000","last_unstale":"2024-05-03T17:21:07.595627+0000","last_undegraded":"2024-05-03T17:21:07.595627+0000","last_fullsized":"2024-05-03T17:21:07.595627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:27:47.685727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593658+0000","last_change":"2024-05-03T17:20:33.285337+0000","last_active":"2024-05-03T17:21:07.593658+0000","last_peered":"2024-05-03T17:21:07.593658+0000","last_clean":"2024-05-03T17:21:07.593658+0000","last_became_active":"2024-05-03T17:20:33.285012+0000","last_became_peered":"2024-05-03T17:20:33.285012+0000","last_unstale":"2024-05-03T17:21:07.593658+0000","last_undegraded":"2024-05-03T17:21:07.593658+0000","last_fullsized":"2024-05-03T17:21:07.593658+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:04:44.246674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593083+0000","last_change":"2024-05-03T17:20:37.211216+0000","last_active":"2024-05-03T17:21:07.593083+0000","last_peered":"2024-05-03T17:21:07.593083+0000","last_clean":"2024-05-03T17:21:07.593083+0000","last_became_active":"2024-05-03T17:20:37.210906+0000","last_became_peered":"2024-05-03T17:20:37.210906+0000","last_unstale":"2024-05-03T17:21:07.593083+0000","last_undegraded":"2024-05-03T17:21:07.593083+0000","last_fullsized":"2024-05-03T17:21:07.593083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:20:45.831133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597030+0000","last_change":"2024-05-03T17:20:33.165952+0000","last_active":"2024-05-03T17:21:07.597030+0000","last_peered":"2024-05-03T17:21:07.597030+0000","last_clean":"2024-05-03T17:21:07.597030+0000","last_became_active":"2024-05-03T17:20:33.165546+0000","last_became_peered":"2024-05-03T17:20:33.165546+0000","last_unstale":"2024-05-03T17:21:07.597030+0000","last_undegraded":"2024-05-03T17:21:07.597030+0000","last_fullsized":"2024-05-03T17:21:07.597030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:26:44.484883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592776+0000","last_change":"2024-05-03T17:20:37.209107+0000","last_active":"2024-05-03T17:21:07.592776+0000","last_peered":"2024-05-03T17:21:07.592776+0000","last_clean":"2024-05-03T17:21:07.592776+0000","last_became_active":"2024-05-03T17:20:37.208754+0000","last_became_peered":"2024-05-03T17:20:37.208754+0000","last_unstale":"2024-05-03T17:21:07.592776+0000","last_undegraded":"2024-05-03T17:21:07.592776+0000","last_fullsized":"2024-05-03T17:21:07.592776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:47:22.142530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604237+0000","last_change":"2024-05-03T17:20:37.209092+0000","last_active":"2024-05-03T17:21:07.604237+0000","last_peered":"2024-05-03T17:21:07.604237+0000","last_clean":"2024-05-03T17:21:07.604237+0000","last_became_active":"2024-05-03T17:20:37.208618+0000","last_became_peered":"2024-05-03T17:20:37.208618+0000","last_unstale":"2024-05-03T17:21:07.604237+0000","last_undegraded":"2024-05-03T17:21:07.604237+0000","last_fullsized":"2024-05-03T17:21:07.604237+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:11:09.509930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'4","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:22.623941+0000","last_change":"2024-05-03T17:20:33.283985+0000","last_active":"2024-05-03T17:21:22.623941+0000","last_peered":"2024-05-03T17:21:22.623941+0000","last_clean":"2024-05-03T17:21:22.623941+0000","last_became_active":"2024-05-03T17:20:33.283629+0000","last_became_peered":"2024-05-03T17:20:33.283629+0000","last_unstale":"2024-05-03T17:21:22.623941+0000","last_undegraded":"2024-05-03T17:21:22.623941+0000","last_fullsized":"2024-05-03T17:21:22.623941+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:03:08.751017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":26,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592649+0000","last_change":"2024-05-03T17:20:37.209086+0000","last_active":"2024-05-03T17:21:07.592649+0000","last_peered":"2024-05-03T17:21:07.592649+0000","last_clean":"2024-05-03T17:21:07.592649+0000","last_became_active":"2024-05-03T17:20:37.207968+0000","last_became_peered":"2024-05-03T17:20:37.207968+0000","last_unstale":"2024-05-03T17:21:07.592649+0000","last_undegraded":"2024-05-03T17:21:07.592649+0000","last_fullsized":"2024-05-03T17:21:07.592649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:57:51.816995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.602944+0000","last_change":"2024-05-03T17:20:33.164302+0000","last_active":"2024-05-03T17:21:07.602944+0000","last_peered":"2024-05-03T17:21:07.602944+0000","last_clean":"2024-05-03T17:21:07.602944+0000","last_became_active":"2024-05-03T17:20:33.163958+0000","last_became_peered":"2024-05-03T17:20:33.163958+0000","last_unstale":"2024-05-03T17:21:07.602944+0000","last_undegraded":"2024-05-03T17:21:07.602944+0000","last_fullsized":"2024-05-03T17:21:07.602944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:09:08.221659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":100,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596315+0000","last_change":"2024-05-03T17:19:04.188548+0000","last_active":"2024-05-03T17:21:07.596315+0000","last_peered":"2024-05-03T17:21:07.596315+0000","last_clean":"2024-05-03T17:21:07.596315+0000","last_became_active":"2024-05-03T17:19:04.183986+0000","last_became_peered":"2024-05-03T17:19:04.183986+0000","last_unstale":"2024-05-03T17:21:07.596315+0000","last_undegraded":"2024-05-03T17:21:07.596315+0000","last_fullsized":"2024-05-03T17:21:07.596315+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:15:37.276361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:15:37.276361+0000","last_clean_scrub_stamp":"2024-05-03T17:15:37.276361+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:45:57.109987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594820+0000","last_change":"2024-05-03T17:20:37.206686+0000","last_active":"2024-05-03T17:21:07.594820+0000","last_peered":"2024-05-03T17:21:07.594820+0000","last_clean":"2024-05-03T17:21:07.594820+0000","last_became_active":"2024-05-03T17:20:37.204810+0000","last_became_peered":"2024-05-03T17:20:37.204810+0000","last_unstale":"2024-05-03T17:21:07.594820+0000","last_undegraded":"2024-05-03T17:21:07.594820+0000","last_fullsized":"2024-05-03T17:21:07.594820+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:52:11.363372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596472+0000","last_change":"2024-05-03T17:20:33.168412+0000","last_active":"2024-05-03T17:21:07.596472+0000","last_peered":"2024-05-03T17:21:07.596472+0000","last_clean":"2024-05-03T17:21:07.596472+0000","last_became_active":"2024-05-03T17:20:33.168089+0000","last_became_peered":"2024-05-03T17:20:33.168089+0000","last_unstale":"2024-05-03T17:21:07.596472+0000","last_undegraded":"2024-05-03T17:21:07.596472+0000","last_fullsized":"2024-05-03T17:21:07.596472+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:03:11.983138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.602992+0000","last_change":"2024-05-03T17:21:07.602540+0000","last_active":"2024-05-03T17:21:07.602992+0000","last_peered":"2024-05-03T17:21:07.602992+0000","last_clean":"2024-05-03T17:21:07.602992+0000","last_became_active":"2024-05-03T17:21:07.599329+0000","last_became_peered":"2024-05-03T17:21:07.599329+0000","last_unstale":"2024-05-03T17:21:07.602992+0000","last_undegraded":"2024-05-03T17:21:07.602992+0000","last_fullsized":"2024-05-03T17:21:07.602992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:57:16.367541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594903+0000","last_change":"2024-05-03T17:20:37.206615+0000","last_active":"2024-05-03T17:21:07.594903+0000","last_peered":"2024-05-03T17:21:07.594903+0000","last_clean":"2024-05-03T17:21:07.594903+0000","last_became_active":"2024-05-03T17:20:37.205519+0000","last_became_peered":"2024-05-03T17:20:37.205519+0000","last_unstale":"2024-05-03T17:21:07.594903+0000","last_undegraded":"2024-05-03T17:21:07.594903+0000","last_fullsized":"2024-05-03T17:21:07.594903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T19:08:45.459749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'10","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:18.215255+0000","last_change":"2024-05-03T17:20:33.164933+0000","last_active":"2024-05-03T17:21:18.215255+0000","last_peered":"2024-05-03T17:21:18.215255+0000","last_clean":"2024-05-03T17:21:18.215255+0000","last_became_active":"2024-05-03T17:20:33.164403+0000","last_became_peered":"2024-05-03T17:20:33.164403+0000","last_unstale":"2024-05-03T17:21:18.215255+0000","last_undegraded":"2024-05-03T17:21:18.215255+0000","last_fullsized":"2024-05-03T17:21:18.215255+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:11:21.158371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592505+0000","last_change":"2024-05-03T17:20:37.211145+0000","last_active":"2024-05-03T17:21:07.592505+0000","last_peered":"2024-05-03T17:21:07.592505+0000","last_clean":"2024-05-03T17:21:07.592505+0000","last_became_active":"2024-05-03T17:20:37.210829+0000","last_became_peered":"2024-05-03T17:20:37.210829+0000","last_unstale":"2024-05-03T17:21:07.592505+0000","last_undegraded":"2024-05-03T17:21:07.592505+0000","last_fullsized":"2024-05-03T17:21:07.592505+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:19:23.868001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603017+0000","last_change":"2024-05-03T17:20:33.166287+0000","last_active":"2024-05-03T17:21:07.603017+0000","last_peered":"2024-05-03T17:21:07.603017+0000","last_clean":"2024-05-03T17:21:07.603017+0000","last_became_active":"2024-05-03T17:20:33.165929+0000","last_became_peered":"2024-05-03T17:20:33.165929+0000","last_unstale":"2024-05-03T17:21:07.603017+0000","last_undegraded":"2024-05-03T17:21:07.603017+0000","last_fullsized":"2024-05-03T17:21:07.603017+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:32:37.093941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599190+0000","last_change":"2024-05-03T17:20:37.186668+0000","last_active":"2024-05-03T17:21:07.599190+0000","last_peered":"2024-05-03T17:21:07.599190+0000","last_clean":"2024-05-03T17:21:07.599190+0000","last_became_active":"2024-05-03T17:20:37.185922+0000","last_became_peered":"2024-05-03T17:20:37.185922+0000","last_unstale":"2024-05-03T17:21:07.599190+0000","last_undegraded":"2024-05-03T17:21:07.599190+0000","last_fullsized":"2024-05-03T17:21:07.599190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:00:11.075950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598383+0000","last_change":"2024-05-03T17:20:33.284515+0000","last_active":"2024-05-03T17:21:07.598383+0000","last_peered":"2024-05-03T17:21:07.598383+0000","last_clean":"2024-05-03T17:21:07.598383+0000","last_became_active":"2024-05-03T17:20:33.284169+0000","last_became_peered":"2024-05-03T17:20:33.284169+0000","last_unstale":"2024-05-03T17:21:07.598383+0000","last_undegraded":"2024-05-03T17:21:07.598383+0000","last_fullsized":"2024-05-03T17:21:07.598383+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:27:49.722159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597482+0000","last_change":"2024-05-03T17:20:37.200724+0000","last_active":"2024-05-03T17:21:07.597482+0000","last_peered":"2024-05-03T17:21:07.597482+0000","last_clean":"2024-05-03T17:21:07.597482+0000","last_became_active":"2024-05-03T17:20:37.200383+0000","last_became_peered":"2024-05-03T17:20:37.200383+0000","last_unstale":"2024-05-03T17:21:07.597482+0000","last_undegraded":"2024-05-03T17:21:07.597482+0000","last_fullsized":"2024-05-03T17:21:07.597482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:12:15.547433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592389+0000","last_change":"2024-05-03T17:20:33.163642+0000","last_active":"2024-05-03T17:21:07.592389+0000","last_peered":"2024-05-03T17:21:07.592389+0000","last_clean":"2024-05-03T17:21:07.592389+0000","last_became_active":"2024-05-03T17:20:33.163270+0000","last_became_peered":"2024-05-03T17:20:33.163270+0000","last_unstale":"2024-05-03T17:21:07.592389+0000","last_undegraded":"2024-05-03T17:21:07.592389+0000","last_fullsized":"2024-05-03T17:21:07.592389+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:48:02.719294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595341+0000","last_change":"2024-05-03T17:20:37.203063+0000","last_active":"2024-05-03T17:21:07.595341+0000","last_peered":"2024-05-03T17:21:07.595341+0000","last_clean":"2024-05-03T17:21:07.595341+0000","last_became_active":"2024-05-03T17:20:37.202609+0000","last_became_peered":"2024-05-03T17:20:37.202609+0000","last_unstale":"2024-05-03T17:21:07.595341+0000","last_undegraded":"2024-05-03T17:21:07.595341+0000","last_fullsized":"2024-05-03T17:21:07.595341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:38:08.497937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593609+0000","last_change":"2024-05-03T17:20:33.168672+0000","last_active":"2024-05-03T17:21:07.593609+0000","last_peered":"2024-05-03T17:21:07.593609+0000","last_clean":"2024-05-03T17:21:07.593609+0000","last_became_active":"2024-05-03T17:20:33.168349+0000","last_became_peered":"2024-05-03T17:20:33.168349+0000","last_unstale":"2024-05-03T17:21:07.593609+0000","last_undegraded":"2024-05-03T17:21:07.593609+0000","last_fullsized":"2024-05-03T17:21:07.593609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:26:04.340040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592378+0000","last_change":"2024-05-03T17:20:37.203725+0000","last_active":"2024-05-03T17:21:07.592378+0000","last_peered":"2024-05-03T17:21:07.592378+0000","last_clean":"2024-05-03T17:21:07.592378+0000","last_became_active":"2024-05-03T17:20:37.203318+0000","last_became_peered":"2024-05-03T17:20:37.203318+0000","last_unstale":"2024-05-03T17:21:07.592378+0000","last_undegraded":"2024-05-03T17:21:07.592378+0000","last_fullsized":"2024-05-03T17:21:07.592378+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:00:48.706036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.609823+0000","last_change":"2024-05-03T17:21:07.609392+0000","last_active":"2024-05-03T17:21:07.609823+0000","last_peered":"2024-05-03T17:21:07.609823+0000","last_clean":"2024-05-03T17:21:07.609823+0000","last_became_active":"2024-05-03T17:21:07.600420+0000","last_became_peered":"2024-05-03T17:21:07.600420+0000","last_unstale":"2024-05-03T17:21:07.609823+0000","last_undegraded":"2024-05-03T17:21:07.609823+0000","last_fullsized":"2024-05-03T17:21:07.609823+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:03:47.869058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597082+0000","last_change":"2024-05-03T17:20:37.203766+0000","last_active":"2024-05-03T17:21:07.597082+0000","last_peered":"2024-05-03T17:21:07.597082+0000","last_clean":"2024-05-03T17:21:07.597082+0000","last_became_active":"2024-05-03T17:20:37.203306+0000","last_became_peered":"2024-05-03T17:20:37.203306+0000","last_unstale":"2024-05-03T17:21:07.597082+0000","last_undegraded":"2024-05-03T17:21:07.597082+0000","last_fullsized":"2024-05-03T17:21:07.597082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:57:22.760234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594059+0000","last_change":"2024-05-03T17:20:33.285014+0000","last_active":"2024-05-03T17:21:07.594059+0000","last_peered":"2024-05-03T17:21:07.594059+0000","last_clean":"2024-05-03T17:21:07.594059+0000","last_became_active":"2024-05-03T17:20:33.284178+0000","last_became_peered":"2024-05-03T17:20:33.284178+0000","last_unstale":"2024-05-03T17:21:07.594059+0000","last_undegraded":"2024-05-03T17:21:07.594059+0000","last_fullsized":"2024-05-03T17:21:07.594059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:45:57.315989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597013+0000","last_change":"2024-05-03T17:20:37.201286+0000","last_active":"2024-05-03T17:21:07.597013+0000","last_peered":"2024-05-03T17:21:07.597013+0000","last_clean":"2024-05-03T17:21:07.597013+0000","last_became_active":"2024-05-03T17:20:37.200879+0000","last_became_peered":"2024-05-03T17:20:37.200879+0000","last_unstale":"2024-05-03T17:21:07.597013+0000","last_undegraded":"2024-05-03T17:21:07.597013+0000","last_fullsized":"2024-05-03T17:21:07.597013+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:25:49.936431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604390+0000","last_change":"2024-05-03T17:20:33.165850+0000","last_active":"2024-05-03T17:21:07.604390+0000","last_peered":"2024-05-03T17:21:07.604390+0000","last_clean":"2024-05-03T17:21:07.604390+0000","last_became_active":"2024-05-03T17:20:33.165478+0000","last_became_peered":"2024-05-03T17:20:33.165478+0000","last_unstale":"2024-05-03T17:21:07.604390+0000","last_undegraded":"2024-05-03T17:21:07.604390+0000","last_fullsized":"2024-05-03T17:21:07.604390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:22:36.224538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596692+0000","last_change":"2024-05-03T17:20:37.210896+0000","last_active":"2024-05-03T17:21:07.596692+0000","last_peered":"2024-05-03T17:21:07.596692+0000","last_clean":"2024-05-03T17:21:07.596692+0000","last_became_active":"2024-05-03T17:20:37.209150+0000","last_became_peered":"2024-05-03T17:20:37.209150+0000","last_unstale":"2024-05-03T17:21:07.596692+0000","last_undegraded":"2024-05-03T17:21:07.596692+0000","last_fullsized":"2024-05-03T17:21:07.596692+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:17:01.870395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594502+0000","last_change":"2024-05-03T17:20:33.169631+0000","last_active":"2024-05-03T17:21:07.594502+0000","last_peered":"2024-05-03T17:21:07.594502+0000","last_clean":"2024-05-03T17:21:07.594502+0000","last_became_active":"2024-05-03T17:20:33.169293+0000","last_became_peered":"2024-05-03T17:20:33.169293+0000","last_unstale":"2024-05-03T17:21:07.594502+0000","last_undegraded":"2024-05-03T17:21:07.594502+0000","last_fullsized":"2024-05-03T17:21:07.594502+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:20:29.988150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597243+0000","last_change":"2024-05-03T17:20:37.201347+0000","last_active":"2024-05-03T17:21:07.597243+0000","last_peered":"2024-05-03T17:21:07.597243+0000","last_clean":"2024-05-03T17:21:07.597243+0000","last_became_active":"2024-05-03T17:20:37.200999+0000","last_became_peered":"2024-05-03T17:20:37.200999+0000","last_unstale":"2024-05-03T17:21:07.597243+0000","last_undegraded":"2024-05-03T17:21:07.597243+0000","last_fullsized":"2024-05-03T17:21:07.597243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:34:23.027957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595391+0000","last_change":"2024-05-03T17:20:37.206379+0000","last_active":"2024-05-03T17:21:07.595391+0000","last_peered":"2024-05-03T17:21:07.595391+0000","last_clean":"2024-05-03T17:21:07.595391+0000","last_became_active":"2024-05-03T17:20:37.206048+0000","last_became_peered":"2024-05-03T17:20:37.206048+0000","last_unstale":"2024-05-03T17:21:07.595391+0000","last_undegraded":"2024-05-03T17:21:07.595391+0000","last_fullsized":"2024-05-03T17:21:07.595391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:46:02.486170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593768+0000","last_change":"2024-05-03T17:20:37.205997+0000","last_active":"2024-05-03T17:21:07.593768+0000","last_peered":"2024-05-03T17:21:07.593768+0000","last_clean":"2024-05-03T17:21:07.593768+0000","last_became_active":"2024-05-03T17:20:37.205602+0000","last_became_peered":"2024-05-03T17:20:37.205602+0000","last_unstale":"2024-05-03T17:21:07.593768+0000","last_undegraded":"2024-05-03T17:21:07.593768+0000","last_fullsized":"2024-05-03T17:21:07.593768+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:16:04.929294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593598+0000","last_change":"2024-05-03T17:20:37.199994+0000","last_active":"2024-05-03T17:21:07.593598+0000","last_peered":"2024-05-03T17:21:07.593598+0000","last_clean":"2024-05-03T17:21:07.593598+0000","last_became_active":"2024-05-03T17:20:37.199653+0000","last_became_peered":"2024-05-03T17:20:37.199653+0000","last_unstale":"2024-05-03T17:21:07.593598+0000","last_undegraded":"2024-05-03T17:21:07.593598+0000","last_fullsized":"2024-05-03T17:21:07.593598+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:23:24.945036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598001+0000","last_change":"2024-05-03T17:20:37.198701+0000","last_active":"2024-05-03T17:21:07.598001+0000","last_peered":"2024-05-03T17:21:07.598001+0000","last_clean":"2024-05-03T17:21:07.598001+0000","last_became_active":"2024-05-03T17:20:37.198374+0000","last_became_peered":"2024-05-03T17:20:37.198374+0000","last_unstale":"2024-05-03T17:21:07.598001+0000","last_undegraded":"2024-05-03T17:21:07.598001+0000","last_fullsized":"2024-05-03T17:21:07.598001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:33:26.744746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599746+0000","last_change":"2024-05-03T17:21:07.599746+0000","last_active":"2024-05-03T17:21:07.599746+0000","last_peered":"2024-05-03T17:21:07.599746+0000","last_clean":"2024-05-03T17:21:07.599746+0000","last_became_active":"2024-05-03T17:21:07.598862+0000","last_became_peered":"2024-05-03T17:21:07.598862+0000","last_unstale":"2024-05-03T17:21:07.599746+0000","last_undegraded":"2024-05-03T17:21:07.599746+0000","last_fullsized":"2024-05-03T17:21:07.599746+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:46:31.214548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596253+0000","last_change":"2024-05-03T17:20:37.207194+0000","last_active":"2024-05-03T17:21:07.596253+0000","last_peered":"2024-05-03T17:21:07.596253+0000","last_clean":"2024-05-03T17:21:07.596253+0000","last_became_active":"2024-05-03T17:20:37.206833+0000","last_became_peered":"2024-05-03T17:20:37.206833+0000","last_unstale":"2024-05-03T17:21:07.596253+0000","last_undegraded":"2024-05-03T17:21:07.596253+0000","last_fullsized":"2024-05-03T17:21:07.596253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:29:32.212841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594069+0000","last_change":"2024-05-03T17:20:37.203613+0000","last_active":"2024-05-03T17:21:07.594069+0000","last_peered":"2024-05-03T17:21:07.594069+0000","last_clean":"2024-05-03T17:21:07.594069+0000","last_became_active":"2024-05-03T17:20:37.203265+0000","last_became_peered":"2024-05-03T17:20:37.203265+0000","last_unstale":"2024-05-03T17:21:07.594069+0000","last_undegraded":"2024-05-03T17:21:07.594069+0000","last_fullsized":"2024-05-03T17:21:07.594069+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:12:35.816804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592409+0000","last_change":"2024-05-03T17:20:37.206208+0000","last_active":"2024-05-03T17:21:07.592409+0000","last_peered":"2024-05-03T17:21:07.592409+0000","last_clean":"2024-05-03T17:21:07.592409+0000","last_became_active":"2024-05-03T17:20:37.205859+0000","last_became_peered":"2024-05-03T17:20:37.205859+0000","last_unstale":"2024-05-03T17:21:07.592409+0000","last_undegraded":"2024-05-03T17:21:07.592409+0000","last_fullsized":"2024-05-03T17:21:07.592409+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:57:55.311154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604102+0000","last_change":"2024-05-03T17:20:37.197239+0000","last_active":"2024-05-03T17:21:07.604102+0000","last_peered":"2024-05-03T17:21:07.604102+0000","last_clean":"2024-05-03T17:21:07.604102+0000","last_became_active":"2024-05-03T17:20:37.196903+0000","last_became_peered":"2024-05-03T17:20:37.196903+0000","last_unstale":"2024-05-03T17:21:07.604102+0000","last_undegraded":"2024-05-03T17:21:07.604102+0000","last_fullsized":"2024-05-03T17:21:07.604102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:29:53.020573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592086+0000","last_change":"2024-05-03T17:20:37.211295+0000","last_active":"2024-05-03T17:21:07.592086+0000","last_peered":"2024-05-03T17:21:07.592086+0000","last_clean":"2024-05-03T17:21:07.592086+0000","last_became_active":"2024-05-03T17:20:37.210939+0000","last_became_peered":"2024-05-03T17:20:37.210939+0000","last_unstale":"2024-05-03T17:21:07.592086+0000","last_undegraded":"2024-05-03T17:21:07.592086+0000","last_fullsized":"2024-05-03T17:21:07.592086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:45:58.770190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598703+0000","last_change":"2024-05-03T17:20:37.209581+0000","last_active":"2024-05-03T17:21:07.598703+0000","last_peered":"2024-05-03T17:21:07.598703+0000","last_clean":"2024-05-03T17:21:07.598703+0000","last_became_active":"2024-05-03T17:20:37.209126+0000","last_became_peered":"2024-05-03T17:20:37.209126+0000","last_unstale":"2024-05-03T17:21:07.598703+0000","last_undegraded":"2024-05-03T17:21:07.598703+0000","last_fullsized":"2024-05-03T17:21:07.598703+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:47:15.485725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595388+0000","last_change":"2024-05-03T17:20:37.204258+0000","last_active":"2024-05-03T17:21:07.595388+0000","last_peered":"2024-05-03T17:21:07.595388+0000","last_clean":"2024-05-03T17:21:07.595388+0000","last_became_active":"2024-05-03T17:20:37.202405+0000","last_became_peered":"2024-05-03T17:20:37.202405+0000","last_unstale":"2024-05-03T17:21:07.595388+0000","last_undegraded":"2024-05-03T17:21:07.595388+0000","last_fullsized":"2024-05-03T17:21:07.595388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:36:35.946607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595536+0000","last_change":"2024-05-03T17:20:37.208909+0000","last_active":"2024-05-03T17:21:07.595536+0000","last_peered":"2024-05-03T17:21:07.595536+0000","last_clean":"2024-05-03T17:21:07.595536+0000","last_became_active":"2024-05-03T17:20:37.208540+0000","last_became_peered":"2024-05-03T17:20:37.208540+0000","last_unstale":"2024-05-03T17:21:07.595536+0000","last_undegraded":"2024-05-03T17:21:07.595536+0000","last_fullsized":"2024-05-03T17:21:07.595536+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:32:41.700487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":213988,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":55834,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":196,"num_read_kb":154,"num_write":159,"num_write_kb":101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":167502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":136,"ondisk_log_size":136,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678042,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":876544,"data_stored":428314,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874271,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":1071606,"data_compressed":42001,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037797,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":428528,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234026,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":428426,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430254,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":1020378,"data_compressed":42001,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593780,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32040,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713112,"statfs":{"total":95995035648,"available":95962226688,"internally_reserved":0,"allocated":876544,"data_stored":427917,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":479543,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":897024,"data_stored":429188,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411401,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":481284,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":892928,"data_stored":428228,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953535,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":880640,"data_stored":428354,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790010,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":1020062,"data_compressed":42001,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34582,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34582,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":51833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":53574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":53411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-03T17:21:30.302 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-03T17:21:30.302 INFO:tasks.cephadm:Setting up client nodes... 2024-05-03T17:21:30.303 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-03T17:21:30.553 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:30.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:30 smithi067 ceph-mon[29731]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:21:31.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:30 smithi082 ceph-mon[33535]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:21:31.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:30 smithi070 ceph-mon[33250]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:21:31.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:31 smithi067 ceph-mon[29731]: pgmap v323: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-03T17:21:31.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:31 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:21:32.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:31 smithi082 ceph-mon[33535]: pgmap v323: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-03T17:21:32.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:31 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:21:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:31 smithi070 ceph-mon[33250]: pgmap v323: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-03T17:21:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:31 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:21:32.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.318+0000 7f3e4889e700 1 -- 172.21.15.67:0/2437157775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e44104750 msgr2=0x7f3e44104bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:32.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.318+0000 7f3e4889e700 1 --2- 172.21.15.67:0/2437157775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e44104750 0x7f3e44104bb0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c0099c0 tx=0x7f3e2c009cd0 comp rx=0 tx=0).stop 2024-05-03T17:21:32.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.318+0000 7f3e4889e700 1 -- 172.21.15.67:0/2437157775 shutdown_connections 2024-05-03T17:21:32.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.318+0000 7f3e4889e700 1 --2- 172.21.15.67:0/2437157775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441050f0 0x7f3e4410b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:32.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.318+0000 7f3e4889e700 1 --2- 172.21.15.67:0/2437157775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3e44103550 0x7f3e44103970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:32.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.318+0000 7f3e4889e700 1 --2- 172.21.15.67:0/2437157775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e44104750 0x7f3e44104bb0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:32.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.318+0000 7f3e4889e700 1 -- 172.21.15.67:0/2437157775 >> 172.21.15.67:0/2437157775 conn(0x7f3e440feaf0 msgr2=0x7f3e44100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:32.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.318+0000 7f3e4889e700 1 -- 172.21.15.67:0/2437157775 shutdown_connections 2024-05-03T17:21:32.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.318+0000 7f3e4889e700 1 -- 172.21.15.67:0/2437157775 wait complete. 2024-05-03T17:21:32.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e4889e700 1 Processor -- start 2024-05-03T17:21:32.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e4889e700 1 -- start start 2024-05-03T17:21:32.322 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e4889e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3e44103550 0x7f3e4419b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e4889e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e44104750 0x7f3e4419bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e4889e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441050f0 0x7f3e441a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e4889e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4410efc0 con 0x7f3e44104750 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e41d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e44104750 0x7f3e4419bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e41d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e44104750 0x7f3e4419bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:42294/0 (socket says 172.21.15.67:42294) 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e41d9b700 1 -- 172.21.15.67:0/660983994 learned_addr learned my addr 172.21.15.67:0/660983994 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e4889e700 1 -- 172.21.15.67:0/660983994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4410ee40 con 0x7f3e441050f0 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e42d9d700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441050f0 0x7f3e441a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e4889e700 1 -- 172.21.15.67:0/660983994 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f3e4410f140 con 0x7f3e44103550 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.319+0000 7f3e4259c700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3e44103550 0x7f3e4419b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.320+0000 7f3e42d9d700 1 -- 172.21.15.67:0/660983994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3e44103550 msgr2=0x7f3e4419b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.320+0000 7f3e42d9d700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3e44103550 0x7f3e4419b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.320+0000 7f3e42d9d700 1 -- 172.21.15.67:0/660983994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e44104750 msgr2=0x7f3e4419bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.320+0000 7f3e42d9d700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e44104750 0x7f3e4419bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:32.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.320+0000 7f3e42d9d700 1 -- 172.21.15.67:0/660983994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e441a3360 con 0x7f3e441050f0 2024-05-03T17:21:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.320+0000 7f3e41d9b700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e44104750 0x7f3e4419bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:21:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.320+0000 7f3e42d9d700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441050f0 0x7f3e441a2cc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00c970 tx=0x7f3e3c00cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.320+0000 7f3e3b7fe700 1 -- 172.21.15.67:0/660983994 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c00e880 con 0x7f3e441050f0 2024-05-03T17:21:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.320+0000 7f3e3b7fe700 1 -- 172.21.15.67:0/660983994 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e3c00f450 con 0x7f3e441050f0 2024-05-03T17:21:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.320+0000 7f3e4889e700 1 -- 172.21.15.67:0/660983994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e441a3650 con 0x7f3e441050f0 2024-05-03T17:21:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.320+0000 7f3e3b7fe700 1 -- 172.21.15.67:0/660983994 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c018640 con 0x7f3e441050f0 2024-05-03T17:21:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.320+0000 7f3e4889e700 1 -- 172.21.15.67:0/660983994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e441a3b90 con 0x7f3e441050f0 2024-05-03T17:21:32.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.321+0000 7f3e4889e700 1 -- 172.21.15.67:0/660983994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e440519e0 con 0x7f3e441050f0 2024-05-03T17:21:32.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.322+0000 7f3e3b7fe700 1 -- 172.21.15.67:0/660983994 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f3e3c00ea20 con 0x7f3e441050f0 2024-05-03T17:21:32.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.323+0000 7f3e3b7fe700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f3e300b2a30 0x7f3e300b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:32.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.323+0000 7f3e4259c700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f3e300b2a30 0x7f3e300b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:32.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.323+0000 7f3e3b7fe700 1 -- 172.21.15.67:0/660983994 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f3e3c0e20c0 con 0x7f3e441050f0 2024-05-03T17:21:32.325 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.324+0000 7f3e4259c700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f3e300b2a30 0x7f3e300b4ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3e34009550 tx=0x7f3e340091b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:32.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.327+0000 7f3e3b7fe700 1 -- 172.21.15.67:0/660983994 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f3e3c014030 con 0x7f3e441050f0 2024-05-03T17:21:32.638 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.636+0000 7f3e4889e700 1 -- 172.21.15.67:0/660983994 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f3e441a4050 con 0x7f3e441050f0 2024-05-03T17:21:32.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.642+0000 7f3e3b7fe700 1 -- 172.21.15.67:0/660983994 <== mon.2 v2:172.21.15.70:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f3e3c0a9cf0 con 0x7f3e441050f0 2024-05-03T17:21:32.643 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2024-05-03T17:21:32.643 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAcHTVmueUZJhAAoWYr3oAtLiAY7CLTaCGr3g== 2024-05-03T17:21:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.644+0000 7f3e4889e700 1 -- 172.21.15.67:0/660983994 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f3e300b2a30 msgr2=0x7f3e300b4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.644+0000 7f3e4889e700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f3e300b2a30 0x7f3e300b4ef0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3e34009550 tx=0x7f3e340091b0 comp rx=0 tx=0).stop 2024-05-03T17:21:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.644+0000 7f3e4889e700 1 -- 172.21.15.67:0/660983994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441050f0 msgr2=0x7f3e441a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.644+0000 7f3e4889e700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441050f0 0x7f3e441a2cc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c00c970 tx=0x7f3e3c00cc80 comp rx=0 tx=0).stop 2024-05-03T17:21:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.644+0000 7f3e4889e700 1 -- 172.21.15.67:0/660983994 shutdown_connections 2024-05-03T17:21:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.644+0000 7f3e4889e700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f3e300b2a30 0x7f3e300b4ef0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.644+0000 7f3e4889e700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f3e441050f0 0x7f3e441a2cc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.644+0000 7f3e4889e700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f3e44103550 0x7f3e4419b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.644+0000 7f3e4889e700 1 --2- 172.21.15.67:0/660983994 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e44104750 0x7f3e4419bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.644+0000 7f3e4889e700 1 -- 172.21.15.67:0/660983994 >> 172.21.15.67:0/660983994 conn(0x7f3e440feaf0 msgr2=0x7f3e440756d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:32.646 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.644+0000 7f3e4889e700 1 -- 172.21.15.67:0/660983994 shutdown_connections 2024-05-03T17:21:32.647 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:32.644+0000 7f3e4889e700 1 -- 172.21.15.67:0/660983994 wait complete. 2024-05-03T17:21:32.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:32 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/660983994' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T17:21:32.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:32 smithi070 ceph-mon[33250]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T17:21:32.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:32 smithi070 ceph-mon[33250]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-03T17:21:32.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:32 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/660983994' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T17:21:32.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:32 smithi067 ceph-mon[29731]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T17:21:32.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:32 smithi067 ceph-mon[29731]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-03T17:21:33.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:32 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/660983994' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T17:21:33.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:32 smithi082 ceph-mon[33535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-03T17:21:33.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:32 smithi082 ceph-mon[33535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-03T17:21:33.410 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:21:33.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-03T17:21:33.410 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-03T17:21:33.450 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-03T17:21:33.450 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-03T17:21:33.450 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph mgr dump --format=json 2024-05-03T17:21:33.902 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:34.287 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:34 smithi067 ceph-mon[29731]: pgmap v324: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-03T17:21:34.287 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:34.287 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:34.287 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:34.287 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:34 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:34.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:34 smithi082 ceph-mon[33535]: pgmap v324: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-03T17:21:34.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:34.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:34.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:34.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:34 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:34 smithi070 ceph-mon[33250]: pgmap v324: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-03T17:21:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:34 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.589+0000 7f87360e0700 1 -- 172.21.15.67:0/3518390755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87301023a0 msgr2=0x7f8730102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.589+0000 7f87360e0700 1 --2- 172.21.15.67:0/3518390755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87301023a0 0x7f8730102820 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8724009990 tx=0x7f8724009ca0 comp rx=0 tx=0).stop 2024-05-03T17:21:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.589+0000 7f87360e0700 1 -- 172.21.15.67:0/3518390755 shutdown_connections 2024-05-03T17:21:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.589+0000 7f87360e0700 1 --2- 172.21.15.67:0/3518390755 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8730102d60 0x7f873010f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.589+0000 7f87360e0700 1 --2- 172.21.15.67:0/3518390755 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8730108410 0x7f87301087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:35.596 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.589+0000 7f87360e0700 1 --2- 172.21.15.67:0/3518390755 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87301023a0 0x7f8730102820 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.589+0000 7f87360e0700 1 -- 172.21.15.67:0/3518390755 >> 172.21.15.67:0/3518390755 conn(0x7f8730079350 msgr2=0x7f8730079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.590+0000 7f87360e0700 1 -- 172.21.15.67:0/3518390755 shutdown_connections 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.590+0000 7f87360e0700 1 -- 172.21.15.67:0/3518390755 wait complete. 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.590+0000 7f87360e0700 1 Processor -- start 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.590+0000 7f87360e0700 1 -- start start 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f87360e0700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87301023a0 0x7f8730104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f87360e0700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8730102d60 0x7f87301045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f87360e0700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8730108410 0x7f8730104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f87360e0700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8730112630 con 0x7f8730108410 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f87360e0700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f87301124b0 con 0x7f87301023a0 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f87360e0700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f87301127b0 con 0x7f8730102d60 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8730108410 0x7f8730104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87301023a0 0x7f8730104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872f7fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87301023a0 0x7f8730104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.67:52592/0 (socket says 172.21.15.67:52592) 2024-05-03T17:21:35.597 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872f7fe700 1 -- 172.21.15.67:0/3282623258 learned_addr learned my addr 172.21.15.67:0/3282623258 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872effd700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8730102d60 0x7f87301045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872ffff700 1 -- 172.21.15.67:0/3282623258 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8730102d60 msgr2=0x7f87301045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872ffff700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8730102d60 0x7f87301045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872ffff700 1 -- 172.21.15.67:0/3282623258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87301023a0 msgr2=0x7f8730104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872ffff700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87301023a0 0x7f8730104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872ffff700 1 -- 172.21.15.67:0/3282623258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87301a81e0 con 0x7f8730108410 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872f7fe700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87301023a0 0x7f8730104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872ffff700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8730108410 0x7f8730104b00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f872800be20 tx=0x7f8728009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872cff9700 1 -- 172.21.15.67:0/3282623258 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8728019070 con 0x7f8730108410 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872cff9700 1 -- 172.21.15.67:0/3282623258 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f872800cb90 con 0x7f8730108410 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f87360e0700 1 -- 172.21.15.67:0/3282623258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87301a84d0 con 0x7f8730108410 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.591+0000 7f872cff9700 1 -- 172.21.15.67:0/3282623258 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87280125a0 con 0x7f8730108410 2024-05-03T17:21:35.598 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.592+0000 7f87360e0700 1 -- 172.21.15.67:0/3282623258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87301a8990 con 0x7f8730108410 2024-05-03T17:21:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.592+0000 7f87360e0700 1 -- 172.21.15.67:0/3282623258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f873006a190 con 0x7f8730108410 2024-05-03T17:21:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.594+0000 7f872cff9700 1 -- 172.21.15.67:0/3282623258 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f872801c070 con 0x7f8730108410 2024-05-03T17:21:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.594+0000 7f872cff9700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f87180b2910 0x7f87180b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.595+0000 7f872f7fe700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f87180b2910 0x7f87180b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.595+0000 7f872cff9700 1 -- 172.21.15.67:0/3282623258 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f87280e1a40 con 0x7f8730108410 2024-05-03T17:21:35.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.595+0000 7f872f7fe700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f87180b2910 0x7f87180b4dd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8720009ea0 tx=0x7f8720009290 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:35.602 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.601+0000 7f872cff9700 1 -- 172.21.15.67:0/3282623258 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f87280a9700 con 0x7f8730108410 2024-05-03T17:21:35.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.894+0000 7f87360e0700 1 -- 172.21.15.67:0/3282623258 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f87301a8c20 con 0x7f8730108410 2024-05-03T17:21:35.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.899+0000 7f872cff9700 1 -- 172.21.15.67:0/3282623258 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+288647 (secure 0 0 0) 0x7f87280a9000 con 0x7f8730108410 2024-05-03T17:21:35.905 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:21:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.903+0000 7f87360e0700 1 -- 172.21.15.67:0/3282623258 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f87180b2910 msgr2=0x7f87180b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.903+0000 7f87360e0700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f87180b2910 0x7f87180b4dd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8720009ea0 tx=0x7f8720009290 comp rx=0 tx=0).stop 2024-05-03T17:21:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.903+0000 7f87360e0700 1 -- 172.21.15.67:0/3282623258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8730108410 msgr2=0x7f8730104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.903+0000 7f87360e0700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8730108410 0x7f8730104b00 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f872800be20 tx=0x7f8728009ed0 comp rx=0 tx=0).stop 2024-05-03T17:21:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.903+0000 7f87360e0700 1 -- 172.21.15.67:0/3282623258 shutdown_connections 2024-05-03T17:21:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.903+0000 7f87360e0700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f87180b2910 0x7f87180b4dd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.903+0000 7f87360e0700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8730108410 0x7f8730104b00 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.903+0000 7f87360e0700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87301023a0 0x7f8730104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.903+0000 7f87360e0700 1 --2- 172.21.15.67:0/3282623258 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8730102d60 0x7f87301045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:35.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.903+0000 7f87360e0700 1 -- 172.21.15.67:0/3282623258 >> 172.21.15.67:0/3282623258 conn(0x7f8730079350 msgr2=0x7f8730101700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:35.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.903+0000 7f87360e0700 1 -- 172.21.15.67:0/3282623258 shutdown_connections 2024-05-03T17:21:35.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:35.903+0000 7f87360e0700 1 -- 172.21.15.67:0/3282623258 wait complete. 2024-05-03T17:21:36.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:36 smithi067 ceph-mon[29731]: pgmap v325: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-03T17:21:36.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:36.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:36 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:36.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:36 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3282623258' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-03T17:21:36.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:36 smithi082 ceph-mon[33535]: pgmap v325: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-03T17:21:36.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:36.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:36 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:36.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:36 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3282623258' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-03T17:21:36.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:36 smithi070 ceph-mon[33250]: pgmap v325: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-03T17:21:36.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:36.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:36 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:36.559 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:36 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3282623258' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-03T17:21:36.611 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":3284262914},{"type":"v1","addr":"172.21.15.67:6801","nonce":3284262914}]},"active_addr":"172.21.15.67:6801/3284262914","active_change":"2024-05-03T17:13:06.324112+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24110,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.67:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1543466086}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":1727110859}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":2300586476}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3058983916}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.67:0","nonce":3656712761}]}]} 2024-05-03T17:21:36.619 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-03T17:21:36.619 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-03T17:21:36.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T17:21:36.886 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:37.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:37 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:21:37.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:37 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:21:37.171 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:37 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:37.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:37 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:21:37.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:37 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:21:37.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:37 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:37.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:37 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:21:37.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:37 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:21:37.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:37 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:21:38.366 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:38 smithi067 ceph-mon[29731]: pgmap v326: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-03T17:21:38.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:38 smithi082 ceph-mon[33535]: pgmap v326: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-03T17:21:38.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:38 smithi070 ceph-mon[33250]: pgmap v326: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-03T17:21:38.613 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.599+0000 7fbe40341700 1 -- 172.21.15.67:0/1443607243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe38102630 msgr2=0x7fbe38102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.599+0000 7fbe40341700 1 --2- 172.21.15.67:0/1443607243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe38102630 0x7fbe38102a10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c0099c0 tx=0x7fbe2c009cd0 comp rx=0 tx=0).stop 2024-05-03T17:21:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.599+0000 7fbe40341700 1 -- 172.21.15.67:0/1443607243 shutdown_connections 2024-05-03T17:21:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.599+0000 7fbe40341700 1 --2- 172.21.15.67:0/1443607243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe381076c0 0x7fbe3810f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.599+0000 7fbe40341700 1 --2- 172.21.15.67:0/1443607243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe38102630 0x7fbe38102a10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.599+0000 7fbe40341700 1 --2- 172.21.15.67:0/1443607243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe38102fe0 0x7fbe38106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.599+0000 7fbe40341700 1 -- 172.21.15.67:0/1443607243 >> 172.21.15.67:0/1443607243 conn(0x7fbe38079350 msgr2=0x7fbe38079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.599+0000 7fbe40341700 1 -- 172.21.15.67:0/1443607243 shutdown_connections 2024-05-03T17:21:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.600+0000 7fbe40341700 1 -- 172.21.15.67:0/1443607243 wait complete. 2024-05-03T17:21:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.600+0000 7fbe40341700 1 Processor -- start 2024-05-03T17:21:38.614 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.600+0000 7fbe40341700 1 -- start start 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.600+0000 7fbe40341700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe38102630 0x7fbe3819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.600+0000 7fbe40341700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe38102fe0 0x7fbe3819fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.600+0000 7fbe40341700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe381076c0 0x7fbe38199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.600+0000 7fbe40341700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fbe38112550 con 0x7fbe38102630 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.600+0000 7fbe40341700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381123d0 con 0x7fbe381076c0 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.600+0000 7fbe40341700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fbe381126d0 con 0x7fbe38102fe0 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.600+0000 7fbe3e0dd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe38102630 0x7fbe3819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.600+0000 7fbe3e0dd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe38102630 0x7fbe3819f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45350/0 (socket says 172.21.15.67:45350) 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.600+0000 7fbe3e0dd700 1 -- 172.21.15.67:0/985454437 learned_addr learned my addr 172.21.15.67:0/985454437 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe3d8dc700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe38102fe0 0x7fbe3819fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe3e0dd700 1 -- 172.21.15.67:0/985454437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe38102fe0 msgr2=0x7fbe3819fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe3e0dd700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe38102fe0 0x7fbe3819fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe3e0dd700 1 -- 172.21.15.67:0/985454437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe381076c0 msgr2=0x7fbe38199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe3e0dd700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe381076c0 0x7fbe38199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe3e0dd700 1 -- 172.21.15.67:0/985454437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe3819a160 con 0x7fbe38102630 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe3d8dc700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe38102fe0 0x7fbe3819fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:21:38.615 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe3e0dd700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe38102630 0x7fbe3819f7f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00b580 tx=0x7fbe2c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe2b7fe700 1 -- 172.21.15.67:0/985454437 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c027070 con 0x7fbe38102630 2024-05-03T17:21:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe40341700 1 -- 172.21.15.67:0/985454437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe3819a3f0 con 0x7fbe38102630 2024-05-03T17:21:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe2b7fe700 1 -- 172.21.15.67:0/985454437 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe2c004020 con 0x7fbe38102630 2024-05-03T17:21:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe2b7fe700 1 -- 172.21.15.67:0/985454437 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe2c022e30 con 0x7fbe38102630 2024-05-03T17:21:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.601+0000 7fbe40341700 1 -- 172.21.15.67:0/985454437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe381a66c0 con 0x7fbe38102630 2024-05-03T17:21:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.602+0000 7fbe40341700 1 -- 172.21.15.67:0/985454437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe3806a190 con 0x7fbe38102630 2024-05-03T17:21:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.604+0000 7fbe2b7fe700 1 -- 172.21.15.67:0/985454437 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fbe2c022590 con 0x7fbe38102630 2024-05-03T17:21:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.604+0000 7fbe2b7fe700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fbe240b2910 0x7fbe240b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.604+0000 7fbe3d8dc700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fbe240b2910 0x7fbe240b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.604+0000 7fbe2b7fe700 1 -- 172.21.15.67:0/985454437 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fbe2c0e34a0 con 0x7fbe38102630 2024-05-03T17:21:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.605+0000 7fbe3d8dc700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fbe240b2910 0x7fbe240b4dd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbe34005950 tx=0x7fbe340058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:38.616 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.608+0000 7fbe2b7fe700 1 -- 172.21.15.67:0/985454437 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fbe2c013070 con 0x7fbe38102630 2024-05-03T17:21:38.823 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.823+0000 7fbe40341700 1 -- 172.21.15.67:0/985454437 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbe3802e510 con 0x7fbe38102630 2024-05-03T17:21:38.825 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.824+0000 7fbe2b7fe700 1 -- 172.21.15.67:0/985454437 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22635 (secure 0 0 0) 0x7fbe2c0ab0e0 con 0x7fbe38102630 2024-05-03T17:21:38.825 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:21:38.825 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":81,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","created":"2024-05-03T17:12:11.709381+0000","modified":"2024-05-03T17:21:07.584731+0000","last_up_change":"2024-05-03T17:19:29.310797+0000","last_in_change":"2024-05-03T17:19:08.848654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T17:15:36.413266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T17:20:31.546843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T17:20:35.261082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80962eb1-a506-49bb-a528-dee035eb4322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6803","nonce":1754365741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6805","nonce":1754365741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6809","nonce":1754365741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6807","nonce":1754365741}]},"public_addr":"172.21.15.67:6803/1754365741","cluster_addr":"172.21.15.67:6805/1754365741","heartbeat_back_addr":"172.21.15.67:6809/1754365741","heartbeat_front_addr":"172.21.15.67:6807/1754365741","state":["exists","up"]},{"osd":1,"uuid":"d7721d5f-4373-4cb2-bb57-bc368d6bb818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6801","nonce":1006339176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6803","nonce":1006339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6807","nonce":1006339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6805","nonce":1006339176}]},"public_addr":"172.21.15.70:6801/1006339176","cluster_addr":"172.21.15.70:6803/1006339176","heartbeat_back_addr":"172.21.15.70:6807/1006339176","heartbeat_front_addr":"172.21.15.70:6805/1006339176","state":["exists","up"]},{"osd":2,"uuid":"9992fd98-821b-40ce-b051-89befbfd87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6801","nonce":1888803264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6803","nonce":1888803264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6807","nonce":1888803264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6805","nonce":1888803264}]},"public_addr":"172.21.15.82:6801/1888803264","cluster_addr":"172.21.15.82:6803/1888803264","heartbeat_back_addr":"172.21.15.82:6807/1888803264","heartbeat_front_addr":"172.21.15.82:6805/1888803264","state":["exists","up"]},{"osd":3,"uuid":"14bfe6a6-0bc8-4766-8d36-fbaecc61a396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6811","nonce":313717723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6813","nonce":313717723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6817","nonce":313717723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6815","nonce":313717723}]},"public_addr":"172.21.15.67:6811/313717723","cluster_addr":"172.21.15.67:6813/313717723","heartbeat_back_addr":"172.21.15.67:6817/313717723","heartbeat_front_addr":"172.21.15.67:6815/313717723","state":["exists","up"]},{"osd":4,"uuid":"5a44aeb8-aabf-4824-be65-2c7170dbee52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6809","nonce":3526300828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6811","nonce":3526300828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6815","nonce":3526300828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6813","nonce":3526300828}]},"public_addr":"172.21.15.70:6809/3526300828","cluster_addr":"172.21.15.70:6811/3526300828","heartbeat_back_addr":"172.21.15.70:6815/3526300828","heartbeat_front_addr":"172.21.15.70:6813/3526300828","state":["exists","up"]},{"osd":5,"uuid":"a6ac56df-466e-4096-8fbf-835f5b544297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6809","nonce":3754223290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6811","nonce":3754223290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6815","nonce":3754223290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6813","nonce":3754223290}]},"public_addr":"172.21.15.82:6809/3754223290","cluster_addr":"172.21.15.82:6811/3754223290","heartbeat_back_addr":"172.21.15.82:6815/3754223290","heartbeat_front_addr":"172.21.15.82:6813/3754223290","state":["exists","up"]},{"osd":6,"uuid":"4d8a5e8a-66ac-462e-b960-24c0ee248fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6819","nonce":4033769161}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6821","nonce":4033769161}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6825","nonce":4033769161}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6823","nonce":4033769161}]},"public_addr":"172.21.15.67:6819/4033769161","cluster_addr":"172.21.15.67:6821/4033769161","heartbeat_back_addr":"172.21.15.67:6825/4033769161","heartbeat_front_addr":"172.21.15.67:6823/4033769161","state":["exists","up"]},{"osd":7,"uuid":"eb983837-8172-4802-9843-6360cda8f4be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6817","nonce":3366146173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6819","nonce":3366146173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6823","nonce":3366146173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6821","nonce":3366146173}]},"public_addr":"172.21.15.70:6817/3366146173","cluster_addr":"172.21.15.70:6819/3366146173","heartbeat_back_addr":"172.21.15.70:6823/3366146173","heartbeat_front_addr":"172.21.15.70:6821/3366146173","state":["exists","up"]},{"osd":8,"uuid":"c9d4ce56-e940-48d1-a9b5-437139e3fed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6817","nonce":347699286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6819","nonce":347699286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6823","nonce":347699286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6821","nonce":347699286}]},"public_addr":"172.21.15.82:6817/347699286","cluster_addr":"172.21.15.82:6819/347699286","heartbeat_back_addr":"172.21.15.82:6823/347699286","heartbeat_front_addr":"172.21.15.82:6821/347699286","state":["exists","up"]},{"osd":9,"uuid":"09596d19-f2d7-485e-a028-9ce5468c6651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6827","nonce":2600095198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6829","nonce":2600095198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6833","nonce":2600095198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6831","nonce":2600095198}]},"public_addr":"172.21.15.67:6827/2600095198","cluster_addr":"172.21.15.67:6829/2600095198","heartbeat_back_addr":"172.21.15.67:6833/2600095198","heartbeat_front_addr":"172.21.15.67:6831/2600095198","state":["exists","up"]},{"osd":10,"uuid":"ec4cffbc-0127-47e5-8176-0972f74a5514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6825","nonce":2897583629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6827","nonce":2897583629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6831","nonce":2897583629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6829","nonce":2897583629}]},"public_addr":"172.21.15.70:6825/2897583629","cluster_addr":"172.21.15.70:6827/2897583629","heartbeat_back_addr":"172.21.15.70:6831/2897583629","heartbeat_front_addr":"172.21.15.70:6829/2897583629","state":["exists","up"]},{"osd":11,"uuid":"a8acfb0b-98e8-49e1-a114-f33aa8160b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6825","nonce":4082656149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6827","nonce":4082656149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6831","nonce":4082656149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6829","nonce":4082656149}]},"public_addr":"172.21.15.82:6825/4082656149","cluster_addr":"172.21.15.82:6827/4082656149","heartbeat_back_addr":"172.21.15.82:6831/4082656149","heartbeat_front_addr":"172.21.15.82:6829/4082656149","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:14:45.596665+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:09.538408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:33.608945+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:59.803102+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:25.187966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:49.892160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:16.575973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:42.115955+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:07.655558+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:35.356779+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:01.144695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:27.577513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/2131289004":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/3335149382":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/992151181":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2614686067":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/3137329076":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2323686292":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/2579128777":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/3335149382":"2024-05-04T17:13:06.324002+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T17:21:38.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.826+0000 7fbe40341700 1 -- 172.21.15.67:0/985454437 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fbe240b2910 msgr2=0x7fbe240b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:38.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.826+0000 7fbe40341700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fbe240b2910 0x7fbe240b4dd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbe34005950 tx=0x7fbe340058e0 comp rx=0 tx=0).stop 2024-05-03T17:21:38.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.826+0000 7fbe40341700 1 -- 172.21.15.67:0/985454437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe38102630 msgr2=0x7fbe3819f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:38.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.826+0000 7fbe40341700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe38102630 0x7fbe3819f7f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbe2c00b580 tx=0x7fbe2c0047b0 comp rx=0 tx=0).stop 2024-05-03T17:21:38.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.826+0000 7fbe40341700 1 -- 172.21.15.67:0/985454437 shutdown_connections 2024-05-03T17:21:38.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.826+0000 7fbe40341700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fbe240b2910 0x7fbe240b4dd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:38.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.826+0000 7fbe40341700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fbe381076c0 0x7fbe38199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:38.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.826+0000 7fbe40341700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fbe38102630 0x7fbe3819f7f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:38.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.826+0000 7fbe40341700 1 --2- 172.21.15.67:0/985454437 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fbe38102fe0 0x7fbe3819fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:38.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.826+0000 7fbe40341700 1 -- 172.21.15.67:0/985454437 >> 172.21.15.67:0/985454437 conn(0x7fbe38079350 msgr2=0x7fbe38106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:38.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.827+0000 7fbe40341700 1 -- 172.21.15.67:0/985454437 shutdown_connections 2024-05-03T17:21:38.828 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:38.827+0000 7fbe40341700 1 -- 172.21.15.67:0/985454437 wait complete. 2024-05-03T17:21:39.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:39 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/985454437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:21:39.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:39 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/985454437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:21:39.549 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-03T17:21:39.550 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T17:21:39.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:39 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/985454437' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:21:39.802 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:40.204 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:40 smithi067 ceph-mon[29731]: pgmap v327: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-03T17:21:40.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:40 smithi082 ceph-mon[33535]: pgmap v327: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-03T17:21:40.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:40 smithi070 ceph-mon[33250]: pgmap v327: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-03T17:21:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.544+0000 7f0588fd7700 1 -- 172.21.15.67:0/479393898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0584108410 msgr2=0x7f05841087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:41.545 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.544+0000 7f0588fd7700 1 --2- 172.21.15.67:0/479393898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0584108410 0x7f05841087f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0574009990 tx=0x7f0574009ca0 comp rx=0 tx=0).stop 2024-05-03T17:21:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.545+0000 7f0588fd7700 1 -- 172.21.15.67:0/479393898 shutdown_connections 2024-05-03T17:21:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.545+0000 7f0588fd7700 1 --2- 172.21.15.67:0/479393898 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0584102d60 0x7f058410f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.545+0000 7f0588fd7700 1 --2- 172.21.15.67:0/479393898 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0584108410 0x7f05841087f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.545+0000 7f0588fd7700 1 --2- 172.21.15.67:0/479393898 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f05841023a0 0x7f0584102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.545+0000 7f0588fd7700 1 -- 172.21.15.67:0/479393898 >> 172.21.15.67:0/479393898 conn(0x7f0584079350 msgr2=0x7f0584079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.545+0000 7f0588fd7700 1 -- 172.21.15.67:0/479393898 shutdown_connections 2024-05-03T17:21:41.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.545+0000 7f0588fd7700 1 -- 172.21.15.67:0/479393898 wait complete. 2024-05-03T17:21:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.545+0000 7f0588fd7700 1 Processor -- start 2024-05-03T17:21:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.545+0000 7f0588fd7700 1 -- start start 2024-05-03T17:21:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0588fd7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f05841023a0 0x7f058419e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0588fd7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0584102d60 0x7f058419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:41.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0588fd7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0584108410 0x7f05841987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0588fd7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0584112530 con 0x7f0584102d60 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0588fd7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f05841123b0 con 0x7f0584108410 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0588fd7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f05841126b0 con 0x7f05841023a0 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0581d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0584102d60 0x7f058419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0581d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0584102d60 0x7f058419ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45376/0 (socket says 172.21.15.67:45376) 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f058259c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f05841023a0 0x7f058419e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0581d9b700 1 -- 172.21.15.67:0/2626353093 learned_addr learned my addr 172.21.15.67:0/2626353093 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0582d9d700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0584108410 0x7f05841987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0581d9b700 1 -- 172.21.15.67:0/2626353093 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f05841023a0 msgr2=0x7f058419e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0581d9b700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f05841023a0 0x7f058419e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0581d9b700 1 -- 172.21.15.67:0/2626353093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0584108410 msgr2=0x7f05841987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0581d9b700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0584108410 0x7f05841987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0581d9b700 1 -- 172.21.15.67:0/2626353093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0584199090 con 0x7f0584102d60 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f0581d9b700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0584102d60 0x7f058419ebd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f057800c9b0 tx=0x7f057800ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:41.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.546+0000 7f05737fe700 1 -- 172.21.15.67:0/2626353093 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578007b00 con 0x7f0584102d60 2024-05-03T17:21:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.547+0000 7f0588fd7700 1 -- 172.21.15.67:0/2626353093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0584199380 con 0x7f0584102d60 2024-05-03T17:21:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.547+0000 7f05737fe700 1 -- 172.21.15.67:0/2626353093 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0578007ca0 con 0x7f0584102d60 2024-05-03T17:21:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.547+0000 7f05737fe700 1 -- 172.21.15.67:0/2626353093 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578018660 con 0x7f0584102d60 2024-05-03T17:21:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.547+0000 7f0588fd7700 1 -- 172.21.15.67:0/2626353093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05841a5610 con 0x7f0584102d60 2024-05-03T17:21:41.550 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.547+0000 7f0588fd7700 1 -- 172.21.15.67:0/2626353093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0584103ac0 con 0x7f0584102d60 2024-05-03T17:21:41.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.549+0000 7f05737fe700 1 -- 172.21.15.67:0/2626353093 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f057800f450 con 0x7f0584102d60 2024-05-03T17:21:41.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.549+0000 7f05737fe700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f056c0b2910 0x7f056c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:41.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.550+0000 7f058259c700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f056c0b2910 0x7f056c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:41.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.550+0000 7f05737fe700 1 -- 172.21.15.67:0/2626353093 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f05780e2650 con 0x7f0584102d60 2024-05-03T17:21:41.551 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.550+0000 7f058259c700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f056c0b2910 0x7f056c0b4dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f057400b5c0 tx=0x7f0574005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:41.553 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.553+0000 7f05737fe700 1 -- 172.21.15.67:0/2626353093 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f0578014030 con 0x7f0584102d60 2024-05-03T17:21:41.774 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.773+0000 7f0588fd7700 1 -- 172.21.15.67:0/2626353093 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0584199c90 con 0x7f0584102d60 2024-05-03T17:21:41.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.775+0000 7f05737fe700 1 -- 172.21.15.67:0/2626353093 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22635 (secure 0 0 0) 0x7f05780aa290 con 0x7f0584102d60 2024-05-03T17:21:41.776 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:21:41.776 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":81,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","created":"2024-05-03T17:12:11.709381+0000","modified":"2024-05-03T17:21:07.584731+0000","last_up_change":"2024-05-03T17:19:29.310797+0000","last_in_change":"2024-05-03T17:19:08.848654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T17:15:36.413266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T17:20:31.546843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T17:20:35.261082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80962eb1-a506-49bb-a528-dee035eb4322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6803","nonce":1754365741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6805","nonce":1754365741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6809","nonce":1754365741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6807","nonce":1754365741}]},"public_addr":"172.21.15.67:6803/1754365741","cluster_addr":"172.21.15.67:6805/1754365741","heartbeat_back_addr":"172.21.15.67:6809/1754365741","heartbeat_front_addr":"172.21.15.67:6807/1754365741","state":["exists","up"]},{"osd":1,"uuid":"d7721d5f-4373-4cb2-bb57-bc368d6bb818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6801","nonce":1006339176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6803","nonce":1006339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6807","nonce":1006339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6805","nonce":1006339176}]},"public_addr":"172.21.15.70:6801/1006339176","cluster_addr":"172.21.15.70:6803/1006339176","heartbeat_back_addr":"172.21.15.70:6807/1006339176","heartbeat_front_addr":"172.21.15.70:6805/1006339176","state":["exists","up"]},{"osd":2,"uuid":"9992fd98-821b-40ce-b051-89befbfd87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6801","nonce":1888803264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6803","nonce":1888803264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6807","nonce":1888803264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6805","nonce":1888803264}]},"public_addr":"172.21.15.82:6801/1888803264","cluster_addr":"172.21.15.82:6803/1888803264","heartbeat_back_addr":"172.21.15.82:6807/1888803264","heartbeat_front_addr":"172.21.15.82:6805/1888803264","state":["exists","up"]},{"osd":3,"uuid":"14bfe6a6-0bc8-4766-8d36-fbaecc61a396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6811","nonce":313717723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6813","nonce":313717723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6817","nonce":313717723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6815","nonce":313717723}]},"public_addr":"172.21.15.67:6811/313717723","cluster_addr":"172.21.15.67:6813/313717723","heartbeat_back_addr":"172.21.15.67:6817/313717723","heartbeat_front_addr":"172.21.15.67:6815/313717723","state":["exists","up"]},{"osd":4,"uuid":"5a44aeb8-aabf-4824-be65-2c7170dbee52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6809","nonce":3526300828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6811","nonce":3526300828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6815","nonce":3526300828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6813","nonce":3526300828}]},"public_addr":"172.21.15.70:6809/3526300828","cluster_addr":"172.21.15.70:6811/3526300828","heartbeat_back_addr":"172.21.15.70:6815/3526300828","heartbeat_front_addr":"172.21.15.70:6813/3526300828","state":["exists","up"]},{"osd":5,"uuid":"a6ac56df-466e-4096-8fbf-835f5b544297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6809","nonce":3754223290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6811","nonce":3754223290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6815","nonce":3754223290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6813","nonce":3754223290}]},"public_addr":"172.21.15.82:6809/3754223290","cluster_addr":"172.21.15.82:6811/3754223290","heartbeat_back_addr":"172.21.15.82:6815/3754223290","heartbeat_front_addr":"172.21.15.82:6813/3754223290","state":["exists","up"]},{"osd":6,"uuid":"4d8a5e8a-66ac-462e-b960-24c0ee248fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6819","nonce":4033769161}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6821","nonce":4033769161}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6825","nonce":4033769161}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6823","nonce":4033769161}]},"public_addr":"172.21.15.67:6819/4033769161","cluster_addr":"172.21.15.67:6821/4033769161","heartbeat_back_addr":"172.21.15.67:6825/4033769161","heartbeat_front_addr":"172.21.15.67:6823/4033769161","state":["exists","up"]},{"osd":7,"uuid":"eb983837-8172-4802-9843-6360cda8f4be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6817","nonce":3366146173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6819","nonce":3366146173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6823","nonce":3366146173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6821","nonce":3366146173}]},"public_addr":"172.21.15.70:6817/3366146173","cluster_addr":"172.21.15.70:6819/3366146173","heartbeat_back_addr":"172.21.15.70:6823/3366146173","heartbeat_front_addr":"172.21.15.70:6821/3366146173","state":["exists","up"]},{"osd":8,"uuid":"c9d4ce56-e940-48d1-a9b5-437139e3fed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6817","nonce":347699286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6819","nonce":347699286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6823","nonce":347699286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6821","nonce":347699286}]},"public_addr":"172.21.15.82:6817/347699286","cluster_addr":"172.21.15.82:6819/347699286","heartbeat_back_addr":"172.21.15.82:6823/347699286","heartbeat_front_addr":"172.21.15.82:6821/347699286","state":["exists","up"]},{"osd":9,"uuid":"09596d19-f2d7-485e-a028-9ce5468c6651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6827","nonce":2600095198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6829","nonce":2600095198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6833","nonce":2600095198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6831","nonce":2600095198}]},"public_addr":"172.21.15.67:6827/2600095198","cluster_addr":"172.21.15.67:6829/2600095198","heartbeat_back_addr":"172.21.15.67:6833/2600095198","heartbeat_front_addr":"172.21.15.67:6831/2600095198","state":["exists","up"]},{"osd":10,"uuid":"ec4cffbc-0127-47e5-8176-0972f74a5514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6825","nonce":2897583629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6827","nonce":2897583629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6831","nonce":2897583629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6829","nonce":2897583629}]},"public_addr":"172.21.15.70:6825/2897583629","cluster_addr":"172.21.15.70:6827/2897583629","heartbeat_back_addr":"172.21.15.70:6831/2897583629","heartbeat_front_addr":"172.21.15.70:6829/2897583629","state":["exists","up"]},{"osd":11,"uuid":"a8acfb0b-98e8-49e1-a114-f33aa8160b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6825","nonce":4082656149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6827","nonce":4082656149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6831","nonce":4082656149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6829","nonce":4082656149}]},"public_addr":"172.21.15.82:6825/4082656149","cluster_addr":"172.21.15.82:6827/4082656149","heartbeat_back_addr":"172.21.15.82:6831/4082656149","heartbeat_front_addr":"172.21.15.82:6829/4082656149","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:14:45.596665+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:09.538408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:33.608945+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:59.803102+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:25.187966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:49.892160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:16.575973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:42.115955+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:07.655558+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:35.356779+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:01.144695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:27.577513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/2131289004":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/3335149382":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/992151181":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2614686067":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/3137329076":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2323686292":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/2579128777":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/3335149382":"2024-05-04T17:13:06.324002+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T17:21:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.777+0000 7f0588fd7700 1 -- 172.21.15.67:0/2626353093 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f056c0b2910 msgr2=0x7f056c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.777+0000 7f0588fd7700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f056c0b2910 0x7f056c0b4dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f057400b5c0 tx=0x7f0574005960 comp rx=0 tx=0).stop 2024-05-03T17:21:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.777+0000 7f0588fd7700 1 -- 172.21.15.67:0/2626353093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0584102d60 msgr2=0x7f058419ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.777+0000 7f0588fd7700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0584102d60 0x7f058419ebd0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f057800c9b0 tx=0x7f057800ccc0 comp rx=0 tx=0).stop 2024-05-03T17:21:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.777+0000 7f0588fd7700 1 -- 172.21.15.67:0/2626353093 shutdown_connections 2024-05-03T17:21:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.777+0000 7f0588fd7700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f056c0b2910 0x7f056c0b4dd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.777+0000 7f0588fd7700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0584108410 0x7f05841987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.777+0000 7f0588fd7700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f05841023a0 0x7f058419e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.777+0000 7f0588fd7700 1 --2- 172.21.15.67:0/2626353093 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0584102d60 0x7f058419ebd0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.777+0000 7f0588fd7700 1 -- 172.21.15.67:0/2626353093 >> 172.21.15.67:0/2626353093 conn(0x7f0584079350 msgr2=0x7f0584100fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:41.778 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.777+0000 7f0588fd7700 1 -- 172.21.15.67:0/2626353093 shutdown_connections 2024-05-03T17:21:41.779 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:41.777+0000 7f0588fd7700 1 -- 172.21.15.67:0/2626353093 wait complete. 2024-05-03T17:21:42.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:42 smithi067 ceph-mon[29731]: pgmap v328: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:21:42.509 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:42 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2626353093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:21:42.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:42 smithi082 ceph-mon[33535]: pgmap v328: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:21:42.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:42 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2626353093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:21:42.534 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-03T17:21:42.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-03T17:21:42.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-03T17:21:42.535 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-03T17:21:42.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-03T17:21:42.536 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-03T17:21:42.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-03T17:21:42.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-03T17:21:42.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-03T17:21:42.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-03T17:21:42.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-03T17:21:42.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-03T17:21:42.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:42 smithi070 ceph-mon[33250]: pgmap v328: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:21:42.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:42 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2626353093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:21:43.210 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:43.213 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:43.296 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:44.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:44 smithi067 ceph-mon[29731]: pgmap v329: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:21:44.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:44 smithi082 ceph-mon[33535]: pgmap v329: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:21:44.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:44 smithi070 ceph-mon[33250]: pgmap v329: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:21:46.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:46 smithi067 ceph-mon[29731]: pgmap v330: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:21:46.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:46 smithi082 ceph-mon[33535]: pgmap v330: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:21:46.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:46 smithi070 ceph-mon[33250]: pgmap v330: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:21:46.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.766+0000 7fa3f59de700 1 -- 172.21.15.67:0/578300788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f006bbd0 msgr2=0x7fa3f006c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:46.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.766+0000 7fa3f59de700 1 --2- 172.21.15.67:0/578300788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f006bbd0 0x7fa3f006c050 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009990 tx=0x7fa3e4009ca0 comp rx=0 tx=0).stop 2024-05-03T17:21:46.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.766+0000 7fa3f59de700 1 -- 172.21.15.67:0/578300788 shutdown_connections 2024-05-03T17:21:46.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.766+0000 7fa3f59de700 1 --2- 172.21.15.67:0/578300788 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa3f006c590 0x7fa3f0110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:46.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.766+0000 7fa3f59de700 1 --2- 172.21.15.67:0/578300788 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3f0109510 0x7fa3f01098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:46.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.766+0000 7fa3f59de700 1 --2- 172.21.15.67:0/578300788 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f006bbd0 0x7fa3f006c050 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:46.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.766+0000 7fa3f59de700 1 -- 172.21.15.67:0/578300788 >> 172.21.15.67:0/578300788 conn(0x7fa3f0079350 msgr2=0x7fa3f0079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:46.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.766+0000 7fa3f59de700 1 -- 172.21.15.67:0/578300788 shutdown_connections 2024-05-03T17:21:46.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.766+0000 7fa3f59de700 1 -- 172.21.15.67:0/578300788 wait complete. 2024-05-03T17:21:46.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.767+0000 7fa3f59de700 1 Processor -- start 2024-05-03T17:21:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.767+0000 7fa3f59de700 1 -- start start 2024-05-03T17:21:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.767+0000 7fa3f59de700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa3f006bbd0 0x7fa3f019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.767+0000 7fa3f59de700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f006c590 0x7fa3f019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.767+0000 7fa3f59de700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3f0109510 0x7fa3f0199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.767+0000 7fa3f59de700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f0113690 con 0x7fa3f006c590 2024-05-03T17:21:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.767+0000 7fa3f59de700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f0113510 con 0x7fa3f0109510 2024-05-03T17:21:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.767+0000 7fa3f59de700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fa3f0113810 con 0x7fa3f006bbd0 2024-05-03T17:21:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.767+0000 7fa3ee7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f006c590 0x7fa3f019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.767+0000 7fa3ee7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f006c590 0x7fa3f019fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43946/0 (socket says 172.21.15.67:43946) 2024-05-03T17:21:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.767+0000 7fa3ee7fc700 1 -- 172.21.15.67:0/3048379713 learned_addr learned my addr 172.21.15.67:0/3048379713 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3eeffd700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa3f006bbd0 0x7fa3f019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:46.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3ef7fe700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3f0109510 0x7fa3f0199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3ee7fc700 1 -- 172.21.15.67:0/3048379713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa3f006bbd0 msgr2=0x7fa3f019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3ee7fc700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fa3f006bbd0 0x7fa3f019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3ee7fc700 1 -- 172.21.15.67:0/3048379713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3f0109510 msgr2=0x7fa3f0199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3ee7fc700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3f0109510 0x7fa3f0199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3ee7fc700 1 -- 172.21.15.67:0/3048379713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3f019a0f0 con 0x7fa3f006c590 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3ef7fe700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fa3f0109510 0x7fa3f0199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3ee7fc700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f006c590 0x7fa3f019fcc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4005910 tx=0x7fa3e40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3f49dc700 1 -- 172.21.15.67:0/3048379713 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4027070 con 0x7fa3f006c590 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3f59de700 1 -- 172.21.15.67:0/3048379713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3f019a380 con 0x7fa3f006c590 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3f49dc700 1 -- 172.21.15.67:0/3048379713 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3e4003f20 con 0x7fa3f006c590 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3f49dc700 1 -- 172.21.15.67:0/3048379713 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4022c70 con 0x7fa3f006c590 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.768+0000 7fa3f59de700 1 -- 172.21.15.67:0/3048379713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3f01a6650 con 0x7fa3f006c590 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.769+0000 7fa3f59de700 1 -- 172.21.15.67:0/3048379713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa3f0063c30 con 0x7fa3f006c590 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.772+0000 7fa3f49dc700 1 -- 172.21.15.67:0/3048379713 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fa3e402a070 con 0x7fa3f006c590 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.773+0000 7fa3f49dc700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa3d80b2910 0x7fa3d80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:46.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.773+0000 7fa3eeffd700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa3d80b2910 0x7fa3d80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.773+0000 7fa3f49dc700 1 -- 172.21.15.67:0/3048379713 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fa3e40e3fb0 con 0x7fa3f006c590 2024-05-03T17:21:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.773+0000 7fa3f49dc700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173] conn(0x7fa3d80bc4d0 0x7fa3d80be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.773+0000 7fa3f49dc700 1 -- 172.21.15.67:0/3048379713 --> [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3e4014e50 con 0x7fa3d80bc4d0 2024-05-03T17:21:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.773+0000 7fa3f49dc700 1 -- 172.21.15.67:0/3048379713 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fa3e40e4400 con 0x7fa3f006c590 2024-05-03T17:21:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.773+0000 7fa3eeffd700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa3d80b2910 0x7fa3d80b4dd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0100a00 tx=0x7fa3e000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.774+0000 7fa3ef7fe700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173] conn(0x7fa3d80bc4d0 0x7fa3d80be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.774+0000 7fa3ef7fe700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173] conn(0x7fa3d80bc4d0 0x7fa3d80be970 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:46.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.775+0000 7fa3f49dc700 1 -- 172.21.15.67:0/3048379713 <== osd.7 v2:172.21.15.70:6816/3366146173 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3e4014e50 con 0x7fa3d80bc4d0 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.792+0000 7fa3f59de700 1 -- 172.21.15.67:0/3048379713 --> [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa3f0064030 con 0x7fa3d80bc4d0 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.793+0000 7fa3f49dc700 1 -- 172.21.15.67:0/3048379713 <== osd.7 v2:172.21.15.70:6816/3366146173 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa3f0064030 con 0x7fa3d80bc4d0 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.793+0000 7fa3f59de700 1 -- 172.21.15.67:0/3048379713 >> [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173] conn(0x7fa3d80bc4d0 msgr2=0x7fa3d80be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.793+0000 7fa3f59de700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173] conn(0x7fa3d80bc4d0 0x7fa3d80be970 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.793+0000 7fa3f59de700 1 -- 172.21.15.67:0/3048379713 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa3d80b2910 msgr2=0x7fa3d80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.793+0000 7fa3f59de700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fa3d80b2910 0x7fa3d80b4dd0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa3f0100a00 tx=0x7fa3e000a380 comp rx=0 tx=0).stop 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.793+0000 7fa3f59de700 1 -- 172.21.15.67:0/3048379713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f006c590 msgr2=0x7fa3f019fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.793+0000 7fa3f59de700 1 --2- 172.21.15.67:0/3048379713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa3f006c590 0x7fa3f019fcc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4005910 tx=0x7fa3e40047b0 comp rx=0 tx=0).stop 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.793+0000 7fa3ef7fe700 1 -- 172.21.15.67:0/3048379713 reap_dead start 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.793+0000 7fa3f59de700 1 -- 172.21.15.67:0/3048379713 shutdown_connections 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.793+0000 7fa3f59de700 1 -- 172.21.15.67:0/3048379713 >> 172.21.15.67:0/3048379713 conn(0x7fa3f0079350 msgr2=0x7fa3f01010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.793+0000 7fa3f59de700 1 -- 172.21.15.67:0/3048379713 shutdown_connections 2024-05-03T17:21:46.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:46.793+0000 7fa3f59de700 1 -- 172.21.15.67:0/3048379713 wait complete. 2024-05-03T17:21:47.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:47 smithi067 ceph-mon[29731]: pgmap v331: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:21:47.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:47 smithi082 ceph-mon[33535]: pgmap v331: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:21:47.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:47 smithi070 ceph-mon[33250]: pgmap v331: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:21:47.859 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:47.861 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:47.872 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:47.874 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:47.883 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:47.893 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:47.897 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:47.900 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:48.116 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.114+0000 7f6931b25700 1 -- 172.21.15.67:0/107604830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692c103390 msgr2=0x7f692c107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.114+0000 7f6931b25700 1 --2- 172.21.15.67:0/107604830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692c103390 0x7f692c107000 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f69140099c0 tx=0x7f6914009cd0 comp rx=0 tx=0).stop 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.114+0000 7f6931b25700 1 -- 172.21.15.67:0/107604830 shutdown_connections 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.114+0000 7f6931b25700 1 --2- 172.21.15.67:0/107604830 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f692c107730 0x7f692c10f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.114+0000 7f6931b25700 1 --2- 172.21.15.67:0/107604830 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c1029e0 0x7f692c102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.114+0000 7f6931b25700 1 --2- 172.21.15.67:0/107604830 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692c103390 0x7f692c107000 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.114+0000 7f6931b25700 1 -- 172.21.15.67:0/107604830 >> 172.21.15.67:0/107604830 conn(0x7f692c079350 msgr2=0x7f692c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.114+0000 7f6931b25700 1 -- 172.21.15.67:0/107604830 shutdown_connections 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.114+0000 7f6931b25700 1 -- 172.21.15.67:0/107604830 wait complete. 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.114+0000 7f6931b25700 1 Processor -- start 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.114+0000 7f6931b25700 1 -- start start 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f6931b25700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f692c1029e0 0x7f692c19f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f6931b25700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692c103390 0x7f692c19fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:48.117 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f6931b25700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c107730 0x7f692c199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f6931b25700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f692c1125a0 con 0x7f692c103390 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f6931b25700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f692c112420 con 0x7f692c1029e0 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f6931b25700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f692c112720 con 0x7f692c107730 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f692affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692c103390 0x7f692c19fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f692affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692c103390 0x7f692c19fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43962/0 (socket says 172.21.15.67:43962) 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f692affd700 1 -- 172.21.15.67:0/1569125798 learned_addr learned my addr 172.21.15.67:0/1569125798 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f692bfff700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c107730 0x7f692c199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f692b7fe700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f692c1029e0 0x7f692c19f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f692affd700 1 -- 172.21.15.67:0/1569125798 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c107730 msgr2=0x7f692c199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f692affd700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f692c107730 0x7f692c199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f692affd700 1 -- 172.21.15.67:0/1569125798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f692c1029e0 msgr2=0x7f692c19f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f692affd700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f692c1029e0 0x7f692c19f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f692affd700 1 -- 172.21.15.67:0/1569125798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692c19a1b0 con 0x7f692c103390 2024-05-03T17:21:48.118 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.115+0000 7f692b7fe700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f692c1029e0 0x7f692c19f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:21:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.116+0000 7f692affd700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692c103390 0x7f692c19fdd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6914006010 tx=0x7f69140047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.116+0000 7f6928ff9700 1 -- 172.21.15.67:0/1569125798 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6914026070 con 0x7f692c103390 2024-05-03T17:21:48.119 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.116+0000 7f6931b25700 1 -- 172.21.15.67:0/1569125798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692c19a440 con 0x7f692c103390 2024-05-03T17:21:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.116+0000 7f6928ff9700 1 -- 172.21.15.67:0/1569125798 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6914003f20 con 0x7f692c103390 2024-05-03T17:21:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.116+0000 7f6928ff9700 1 -- 172.21.15.67:0/1569125798 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f691400fc90 con 0x7f692c103390 2024-05-03T17:21:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.116+0000 7f6931b25700 1 -- 172.21.15.67:0/1569125798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f692c1a6760 con 0x7f692c103390 2024-05-03T17:21:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.117+0000 7f6931b25700 1 -- 172.21.15.67:0/1569125798 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f692c063c30 con 0x7f692c103390 2024-05-03T17:21:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.118+0000 7f6928ff9700 1 -- 172.21.15.67:0/1569125798 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f6914029070 con 0x7f692c103390 2024-05-03T17:21:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.119+0000 7f6928ff9700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f69180b2910 0x7f69180b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:48.120 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.119+0000 7f692b7fe700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f69180b2910 0x7f69180b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.119+0000 7f6928ff9700 1 -- 172.21.15.67:0/1569125798 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f69140e33e0 con 0x7f692c103390 2024-05-03T17:21:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.119+0000 7f6928ff9700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741] conn(0x7f69180bc4d0 0x7f69180be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.119+0000 7f6928ff9700 1 -- 172.21.15.67:0/1569125798 --> [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f691400b0b0 con 0x7f69180bc4d0 2024-05-03T17:21:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.119+0000 7f6928ff9700 1 -- 172.21.15.67:0/1569125798 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f69140e3830 con 0x7f692c103390 2024-05-03T17:21:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.120+0000 7f692bfff700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741] conn(0x7f69180bc4d0 0x7f69180be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.120+0000 7f692b7fe700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f69180b2910 0x7f69180b4dd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f691c0060b0 tx=0x7f691c0076c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.120+0000 7f692bfff700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741] conn(0x7f69180bc4d0 0x7f69180be970 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:48.121 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.121+0000 7f6928ff9700 1 -- 172.21.15.67:0/1569125798 <== osd.0 v2:172.21.15.67:6802/1754365741 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f691400b0b0 con 0x7f69180bc4d0 2024-05-03T17:21:48.122 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:21:48.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.150+0000 7f6931b25700 1 -- 172.21.15.67:0/1569125798 --> [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f692c064030 con 0x7f69180bc4d0 2024-05-03T17:21:48.151 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.150+0000 7f6928ff9700 1 -- 172.21.15.67:0/1569125798 <== osd.0 v2:172.21.15.67:6802/1754365741 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f692c064030 con 0x7f69180bc4d0 2024-05-03T17:21:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.151+0000 7f6931b25700 1 -- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741] conn(0x7f69180bc4d0 msgr2=0x7f69180be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.151+0000 7f6931b25700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741] conn(0x7f69180bc4d0 0x7f69180be970 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.151+0000 7f6931b25700 1 -- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f69180b2910 msgr2=0x7f69180b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.151+0000 7f6931b25700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f69180b2910 0x7f69180b4dd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f691c0060b0 tx=0x7f691c0076c0 comp rx=0 tx=0).stop 2024-05-03T17:21:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.151+0000 7f6931b25700 1 -- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692c103390 msgr2=0x7f692c19fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.151+0000 7f6931b25700 1 --2- 172.21.15.67:0/1569125798 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f692c103390 0x7f692c19fdd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f6914006010 tx=0x7f69140047b0 comp rx=0 tx=0).stop 2024-05-03T17:21:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.151+0000 7f692bfff700 1 -- 172.21.15.67:0/1569125798 reap_dead start 2024-05-03T17:21:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.151+0000 7f6931b25700 1 -- 172.21.15.67:0/1569125798 shutdown_connections 2024-05-03T17:21:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.151+0000 7f6931b25700 1 -- 172.21.15.67:0/1569125798 >> 172.21.15.67:0/1569125798 conn(0x7f692c079350 msgr2=0x7f692c06c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:48.152 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.151+0000 7f6931b25700 1 -- 172.21.15.67:0/1569125798 shutdown_connections 2024-05-03T17:21:48.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:48.151+0000 7f6931b25700 1 -- 172.21.15.67:0/1569125798 wait complete. 2024-05-03T17:21:48.625 INFO:teuthology.orchestra.run.smithi067.stdout:206158430259 2024-05-03T17:21:48.625 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-03T17:21:49.674 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:49 smithi067 ceph-mon[29731]: pgmap v332: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:21:49.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:49 smithi082 ceph-mon[33535]: pgmap v332: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:21:49.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:49 smithi070 ceph-mon[33250]: pgmap v332: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:21:51.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:51 smithi067 ceph-mon[29731]: pgmap v333: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:21:51.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:51 smithi082 ceph-mon[33535]: pgmap v333: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:21:51.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:51 smithi070 ceph-mon[33250]: pgmap v333: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:21:53.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:53 smithi067 ceph-mon[29731]: pgmap v334: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:21:53.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:53 smithi082 ceph-mon[33535]: pgmap v334: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:21:53.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:53 smithi070 ceph-mon[33250]: pgmap v334: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:21:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.881+0000 7fb3cfd90700 1 -- 172.21.15.67:0/658700375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c8102630 msgr2=0x7fb3c8102a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.881+0000 7fb3cfd90700 1 --2- 172.21.15.67:0/658700375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c8102630 0x7fb3c8102a10 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fb3c40099c0 tx=0x7fb3c4009cd0 comp rx=0 tx=0).stop 2024-05-03T17:21:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.881+0000 7fb3cfd90700 1 -- 172.21.15.67:0/658700375 shutdown_connections 2024-05-03T17:21:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.881+0000 7fb3cfd90700 1 --2- 172.21.15.67:0/658700375 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c81076c0 0x7fb3c810f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.881+0000 7fb3cfd90700 1 --2- 172.21.15.67:0/658700375 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c8102630 0x7fb3c8102a10 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.881+0000 7fb3cfd90700 1 --2- 172.21.15.67:0/658700375 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3c8102fe0 0x7fb3c8106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:53.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.881+0000 7fb3cfd90700 1 -- 172.21.15.67:0/658700375 >> 172.21.15.67:0/658700375 conn(0x7fb3c8079350 msgr2=0x7fb3c8079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.881+0000 7fb3cfd90700 1 -- 172.21.15.67:0/658700375 shutdown_connections 2024-05-03T17:21:53.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.881+0000 7fb3cfd90700 1 -- 172.21.15.67:0/658700375 wait complete. 2024-05-03T17:21:53.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.882+0000 7fb3cfd90700 1 Processor -- start 2024-05-03T17:21:53.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.882+0000 7fb3cfd90700 1 -- start start 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.882+0000 7fb3cfd90700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c8102630 0x7fb3c819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.882+0000 7fb3cfd90700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c8102fe0 0x7fb3c819fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.882+0000 7fb3cfd90700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3c81076c0 0x7fb3c8199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.882+0000 7fb3cfd90700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c8112570 con 0x7fb3c8102fe0 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.882+0000 7fb3cfd90700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c81123f0 con 0x7fb3c8102630 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.882+0000 7fb3cfd90700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb3c81126f0 con 0x7fb3c81076c0 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.882+0000 7fb3cd32b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c8102fe0 0x7fb3c819fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.882+0000 7fb3cd32b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c8102fe0 0x7fb3c819fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:43982/0 (socket says 172.21.15.67:43982) 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.882+0000 7fb3cd32b700 1 -- 172.21.15.67:0/1605854736 learned_addr learned my addr 172.21.15.67:0/1605854736 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3ce32d700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3c81076c0 0x7fb3c8199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3cdb2c700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c8102630 0x7fb3c819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3cd32b700 1 -- 172.21.15.67:0/1605854736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3c81076c0 msgr2=0x7fb3c8199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3cd32b700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb3c81076c0 0x7fb3c8199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:53.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3cd32b700 1 -- 172.21.15.67:0/1605854736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c8102630 msgr2=0x7fb3c819f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3cd32b700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb3c8102630 0x7fb3c819f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3cd32b700 1 -- 172.21.15.67:0/1605854736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3c819a180 con 0x7fb3c8102fe0 2024-05-03T17:21:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3cd32b700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c8102fe0 0x7fb3c819fda0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb3b800e9f0 tx=0x7fb3b800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3beffd700 1 -- 172.21.15.67:0/1605854736 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b8004a30 con 0x7fb3c8102fe0 2024-05-03T17:21:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3beffd700 1 -- 172.21.15.67:0/1605854736 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3b800cc70 con 0x7fb3c8102fe0 2024-05-03T17:21:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3cfd90700 1 -- 172.21.15.67:0/1605854736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3c819a470 con 0x7fb3c8102fe0 2024-05-03T17:21:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3beffd700 1 -- 172.21.15.67:0/1605854736 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b8005530 con 0x7fb3c8102fe0 2024-05-03T17:21:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.883+0000 7fb3cfd90700 1 -- 172.21.15.67:0/1605854736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3c81a6730 con 0x7fb3c8102fe0 2024-05-03T17:21:53.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.884+0000 7fb3cfd90700 1 -- 172.21.15.67:0/1605854736 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb3c8063bf0 con 0x7fb3c8102fe0 2024-05-03T17:21:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.886+0000 7fb3beffd700 1 -- 172.21.15.67:0/1605854736 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fb3b8004bd0 con 0x7fb3c8102fe0 2024-05-03T17:21:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.886+0000 7fb3beffd700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb3b40b2910 0x7fb3b40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.886+0000 7fb3cdb2c700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb3b40b2910 0x7fb3b40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.886+0000 7fb3beffd700 1 -- 172.21.15.67:0/1605854736 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fb3b80e29e0 con 0x7fb3c8102fe0 2024-05-03T17:21:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.887+0000 7fb3beffd700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198] conn(0x7fb3b40bc4d0 0x7fb3b40be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.887+0000 7fb3beffd700 1 -- 172.21.15.67:0/1605854736 --> [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3b8021e50 con 0x7fb3b40bc4d0 2024-05-03T17:21:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.887+0000 7fb3beffd700 1 -- 172.21.15.67:0/1605854736 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fb3b80e2e30 con 0x7fb3c8102fe0 2024-05-03T17:21:53.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.887+0000 7fb3cdb2c700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb3b40b2910 0x7fb3b40b4dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4000c00 tx=0x7fb3c4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:53.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.887+0000 7fb3ce32d700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198] conn(0x7fb3b40bc4d0 0x7fb3b40be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:53.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.887+0000 7fb3ce32d700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198] conn(0x7fb3b40bc4d0 0x7fb3b40be970 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:53.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.888+0000 7fb3beffd700 1 -- 172.21.15.67:0/1605854736 <== osd.9 v2:172.21.15.67:6826/2600095198 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb3b8021e50 con 0x7fb3b40bc4d0 2024-05-03T17:21:53.906 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.905+0000 7fb3cfd90700 1 -- 172.21.15.67:0/1605854736 --> [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb3c8064070 con 0x7fb3b40bc4d0 2024-05-03T17:21:53.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.906+0000 7fb3beffd700 1 -- 172.21.15.67:0/1605854736 <== osd.9 v2:172.21.15.67:6826/2600095198 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb3c8064070 con 0x7fb3b40bc4d0 2024-05-03T17:21:53.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.907+0000 7fb3cfd90700 1 -- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198] conn(0x7fb3b40bc4d0 msgr2=0x7fb3b40be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:53.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.907+0000 7fb3cfd90700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198] conn(0x7fb3b40bc4d0 0x7fb3b40be970 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:53.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.907+0000 7fb3cfd90700 1 -- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb3b40b2910 msgr2=0x7fb3b40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:53.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.907+0000 7fb3cfd90700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb3b40b2910 0x7fb3b40b4dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4000c00 tx=0x7fb3c4005fb0 comp rx=0 tx=0).stop 2024-05-03T17:21:53.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.907+0000 7fb3cfd90700 1 -- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c8102fe0 msgr2=0x7fb3c819fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:53.908 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.907+0000 7fb3cfd90700 1 --2- 172.21.15.67:0/1605854736 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb3c8102fe0 0x7fb3c819fda0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fb3b800e9f0 tx=0x7fb3b800ed00 comp rx=0 tx=0).stop 2024-05-03T17:21:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.907+0000 7fb3ce32d700 1 -- 172.21.15.67:0/1605854736 reap_dead start 2024-05-03T17:21:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.907+0000 7fb3cfd90700 1 -- 172.21.15.67:0/1605854736 shutdown_connections 2024-05-03T17:21:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.907+0000 7fb3cfd90700 1 -- 172.21.15.67:0/1605854736 >> 172.21.15.67:0/1605854736 conn(0x7fb3c8079350 msgr2=0x7fb3c8106440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.908+0000 7fb3cfd90700 1 -- 172.21.15.67:0/1605854736 shutdown_connections 2024-05-03T17:21:53.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:53.908+0000 7fb3cfd90700 1 -- 172.21.15.67:0/1605854736 wait complete. 2024-05-03T17:21:55.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:55 smithi067 ceph-mon[29731]: pgmap v335: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:21:55.764 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:55 smithi082 ceph-mon[33535]: pgmap v335: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:21:55.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:55 smithi070 ceph-mon[33250]: pgmap v335: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:21:56.354 INFO:teuthology.orchestra.run.smithi067.stdout:257698037803 2024-05-03T17:21:56.354 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-03T17:21:58.231 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:21:58 smithi067 ceph-mon[29731]: pgmap v336: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:21:58.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:21:58 smithi082 ceph-mon[33535]: pgmap v336: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:21:58.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:21:58 smithi070 ceph-mon[33250]: pgmap v336: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:21:58.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.455+0000 7fd69c36e700 1 -- 172.21.15.67:0/185874635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd69406b180 msgr2=0x7fd69406b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.455+0000 7fd69c36e700 1 --2- 172.21.15.67:0/185874635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd69406b180 0x7fd69406b600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd690009990 tx=0x7fd690009ca0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.455+0000 7fd69c36e700 1 -- 172.21.15.67:0/185874635 shutdown_connections 2024-05-03T17:21:58.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.455+0000 7fd69c36e700 1 --2- 172.21.15.67:0/185874635 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd69406bb40 0x7fd694110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.455+0000 7fd69c36e700 1 --2- 172.21.15.67:0/185874635 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd694109510 0x7fd6941098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.455+0000 7fd69c36e700 1 --2- 172.21.15.67:0/185874635 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd69406b180 0x7fd69406b600 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.455+0000 7fd69c36e700 1 -- 172.21.15.67:0/185874635 >> 172.21.15.67:0/185874635 conn(0x7fd694079350 msgr2=0x7fd694079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.455+0000 7fd69c36e700 1 -- 172.21.15.67:0/185874635 shutdown_connections 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.455+0000 7fd69c36e700 1 -- 172.21.15.67:0/185874635 wait complete. 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69c36e700 1 Processor -- start 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69c36e700 1 -- start start 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69c36e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd69406b180 0x7fd69419f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69c36e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd69406bb40 0x7fd69419fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69c36e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd694109510 0x7fd694199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69c36e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd694113640 con 0x7fd694109510 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69c36e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd6941134c0 con 0x7fd69406bb40 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69c36e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd6941137c0 con 0x7fd69406b180 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69a90b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd694109510 0x7fd694199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69a90b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd694109510 0x7fd694199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44048/0 (socket says 172.21.15.67:44048) 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69a90b700 1 -- 172.21.15.67:0/1863953990 learned_addr learned my addr 172.21.15.67:0/1863953990 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69a10a700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd69406b180 0x7fd69419f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd699909700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd69406bb40 0x7fd69419fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69a90b700 1 -- 172.21.15.67:0/1863953990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd69406b180 msgr2=0x7fd69419f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69a90b700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd69406b180 0x7fd69419f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69a90b700 1 -- 172.21.15.67:0/1863953990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd69406bb40 msgr2=0x7fd69419fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69a90b700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd69406bb40 0x7fd69419fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69a90b700 1 -- 172.21.15.67:0/1863953990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69419a0c0 con 0x7fd694109510 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.456+0000 7fd69a90b700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd694109510 0x7fd694199960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd68c007d70 tx=0x7fd68c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.457+0000 7fd6877fe700 1 -- 172.21.15.67:0/1863953990 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68c016020 con 0x7fd694109510 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.457+0000 7fd69c36e700 1 -- 172.21.15.67:0/1863953990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd69419a3b0 con 0x7fd694109510 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.457+0000 7fd6877fe700 1 -- 172.21.15.67:0/1863953990 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd68c00ac90 con 0x7fd694109510 2024-05-03T17:21:58.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.457+0000 7fd6877fe700 1 -- 172.21.15.67:0/1863953990 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd68c0115e0 con 0x7fd694109510 2024-05-03T17:21:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.457+0000 7fd69c36e700 1 -- 172.21.15.67:0/1863953990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6941a6670 con 0x7fd694109510 2024-05-03T17:21:58.461 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.457+0000 7fd69c36e700 1 -- 172.21.15.67:0/1863953990 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd694063c30 con 0x7fd694109510 2024-05-03T17:21:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.459+0000 7fd6877fe700 1 -- 172.21.15.67:0/1863953990 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd68c019020 con 0x7fd694109510 2024-05-03T17:21:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.460+0000 7fd6877fe700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd6800b2910 0x7fd6800b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.460+0000 7fd69a10a700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd6800b2910 0x7fd6800b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.460+0000 7fd6877fe700 1 -- 172.21.15.67:0/1863953990 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fd68c0e26e0 con 0x7fd694109510 2024-05-03T17:21:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.460+0000 7fd6877fe700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286] conn(0x7fd6800bc4d0 0x7fd6800be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.460+0000 7fd6877fe700 1 -- 172.21.15.67:0/1863953990 --> [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6800bf040 con 0x7fd6800bc4d0 2024-05-03T17:21:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.460+0000 7fd6877fe700 1 -- 172.21.15.67:0/1863953990 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fd68c0e2b30 con 0x7fd694109510 2024-05-03T17:21:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.460+0000 7fd69a10a700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd6800b2910 0x7fd6800b4dd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd69419aaa0 tx=0x7fd68800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.460+0000 7fd699909700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286] conn(0x7fd6800bc4d0 0x7fd6800be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.461+0000 7fd699909700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286] conn(0x7fd6800bc4d0 0x7fd6800be970 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:58.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.461+0000 7fd6877fe700 1 -- 172.21.15.67:0/1863953990 <== osd.8 v2:172.21.15.82:6816/347699286 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd6800bf040 con 0x7fd6800bc4d0 2024-05-03T17:21:58.478 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.478+0000 7fd69c36e700 1 -- 172.21.15.67:0/1863953990 --> [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd6940640b0 con 0x7fd6800bc4d0 2024-05-03T17:21:58.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.478+0000 7fd6877fe700 1 -- 172.21.15.67:0/1863953990 <== osd.8 v2:172.21.15.82:6816/347699286 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd6940640b0 con 0x7fd6800bc4d0 2024-05-03T17:21:58.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.479+0000 7fd69c36e700 1 -- 172.21.15.67:0/1863953990 >> [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286] conn(0x7fd6800bc4d0 msgr2=0x7fd6800be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.479+0000 7fd69c36e700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286] conn(0x7fd6800bc4d0 0x7fd6800be970 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.479+0000 7fd69c36e700 1 -- 172.21.15.67:0/1863953990 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd6800b2910 msgr2=0x7fd6800b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.479+0000 7fd69c36e700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd6800b2910 0x7fd6800b4dd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd69419aaa0 tx=0x7fd68800b3c0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.479+0000 7fd69c36e700 1 -- 172.21.15.67:0/1863953990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd694109510 msgr2=0x7fd694199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.479+0000 7fd69c36e700 1 --2- 172.21.15.67:0/1863953990 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd694109510 0x7fd694199960 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd68c007d70 tx=0x7fd68c00a340 comp rx=0 tx=0).stop 2024-05-03T17:21:58.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.479+0000 7fd69a90b700 1 -- 172.21.15.67:0/1863953990 reap_dead start 2024-05-03T17:21:58.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.479+0000 7fd69c36e700 1 -- 172.21.15.67:0/1863953990 shutdown_connections 2024-05-03T17:21:58.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.479+0000 7fd69c36e700 1 -- 172.21.15.67:0/1863953990 >> 172.21.15.67:0/1863953990 conn(0x7fd694079350 msgr2=0x7fd694101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:58.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.479+0000 7fd69c36e700 1 -- 172.21.15.67:0/1863953990 shutdown_connections 2024-05-03T17:21:58.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.479+0000 7fd69c36e700 1 -- 172.21.15.67:0/1863953990 wait complete. 2024-05-03T17:21:58.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.336+0000 7fd2df0a1700 1 -- 172.21.15.67:0/1570403037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d8104100 msgr2=0x7fd2d8108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.336+0000 7fd2df0a1700 1 --2- 172.21.15.67:0/1570403037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d8104100 0x7fd2d8108120 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd2c40099c0 tx=0x7fd2c4009cd0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.336+0000 7fd2df0a1700 1 -- 172.21.15.67:0/1570403037 shutdown_connections 2024-05-03T17:21:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.336+0000 7fd2df0a1700 1 --2- 172.21.15.67:0/1570403037 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2d8108850 0x7fd2d81103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.336+0000 7fd2df0a1700 1 --2- 172.21.15.67:0/1570403037 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2d8103750 0x7fd2d8103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.336+0000 7fd2df0a1700 1 --2- 172.21.15.67:0/1570403037 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d8104100 0x7fd2d8108120 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.336+0000 7fd2df0a1700 1 -- 172.21.15.67:0/1570403037 >> 172.21.15.67:0/1570403037 conn(0x7fd2d807a600 msgr2=0x7fd2d807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.336+0000 7fd2df0a1700 1 -- 172.21.15.67:0/1570403037 shutdown_connections 2024-05-03T17:21:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.336+0000 7fd2df0a1700 1 -- 172.21.15.67:0/1570403037 wait complete. 2024-05-03T17:21:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2df0a1700 1 Processor -- start 2024-05-03T17:21:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2df0a1700 1 -- start start 2024-05-03T17:21:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2df0a1700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2d8103750 0x7fd2d819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2df0a1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d8104100 0x7fd2d819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2df0a1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2d8108850 0x7fd2d8199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2df0a1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d8113630 con 0x7fd2d8104100 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2df0a1700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81134b0 con 0x7fd2d8108850 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2df0a1700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81137b0 con 0x7fd2d8103750 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2d7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d8104100 0x7fd2d819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2d7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d8104100 0x7fd2d819fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44028/0 (socket says 172.21.15.67:44028) 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2d7fff700 1 -- 172.21.15.67:0/509454080 learned_addr learned my addr 172.21.15.67:0/509454080 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2dce3d700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2d8103750 0x7fd2d819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2dd63e700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2d8108850 0x7fd2d8199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2d7fff700 1 -- 172.21.15.67:0/509454080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2d8103750 msgr2=0x7fd2d819f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2d7fff700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2d8103750 0x7fd2d819f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2d7fff700 1 -- 172.21.15.67:0/509454080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2d8108850 msgr2=0x7fd2d8199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.569 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2d7fff700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd2d8108850 0x7fd2d8199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2d7fff700 1 -- 172.21.15.67:0/509454080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d819a070 con 0x7fd2d8104100 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2dce3d700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd2d8103750 0x7fd2d819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.337+0000 7fd2d7fff700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d8104100 0x7fd2d819fc90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400b580 tx=0x7fd2c40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.338+0000 7fd2d5ffb700 1 -- 172.21.15.67:0/509454080 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4027070 con 0x7fd2d8104100 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.338+0000 7fd2d5ffb700 1 -- 172.21.15.67:0/509454080 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd2c4004020 con 0x7fd2d8104100 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.338+0000 7fd2df0a1700 1 -- 172.21.15.67:0/509454080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d819a300 con 0x7fd2d8104100 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.338+0000 7fd2d5ffb700 1 -- 172.21.15.67:0/509454080 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4022e30 con 0x7fd2d8104100 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.338+0000 7fd2df0a1700 1 -- 172.21.15.67:0/509454080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d81a6620 con 0x7fd2d8104100 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.339+0000 7fd2df0a1700 1 -- 172.21.15.67:0/509454080 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd2d8063c30 con 0x7fd2d8104100 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.340+0000 7fd2d5ffb700 1 -- 172.21.15.67:0/509454080 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd2c4022590 con 0x7fd2d8104100 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.341+0000 7fd2d5ffb700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd2c80b2990 0x7fd2c80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.341+0000 7fd2dce3d700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd2c80b2990 0x7fd2c80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.341+0000 7fd2d5ffb700 1 -- 172.21.15.67:0/509454080 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fd2c40e3760 con 0x7fd2d8104100 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.341+0000 7fd2d5ffb700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723] conn(0x7fd2c80bc420 0x7fd2c80be8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.570 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.341+0000 7fd2d5ffb700 1 -- 172.21.15.67:0/509454080 --> [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd2c400b0b0 con 0x7fd2c80bc420 2024-05-03T17:21:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.341+0000 7fd2d5ffb700 1 -- 172.21.15.67:0/509454080 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fd2c40e3bb0 con 0x7fd2d8104100 2024-05-03T17:21:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.341+0000 7fd2dce3d700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd2c80b2990 0x7fd2c80b4e50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd2d819aa50 tx=0x7fd2cc00b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.341+0000 7fd2dd63e700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723] conn(0x7fd2c80bc420 0x7fd2c80be8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.342+0000 7fd2dd63e700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723] conn(0x7fd2c80bc420 0x7fd2c80be8c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.342+0000 7fd2d5ffb700 1 -- 172.21.15.67:0/509454080 <== osd.3 v2:172.21.15.67:6810/313717723 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd2c400b0b0 con 0x7fd2c80bc420 2024-05-03T17:21:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.360+0000 7fd2df0a1700 1 -- 172.21.15.67:0/509454080 --> [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd2d8064030 con 0x7fd2c80bc420 2024-05-03T17:21:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.361+0000 7fd2d5ffb700 1 -- 172.21.15.67:0/509454080 <== osd.3 v2:172.21.15.67:6810/313717723 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd2d8064030 con 0x7fd2c80bc420 2024-05-03T17:21:58.571 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.361+0000 7fd2df0a1700 1 -- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723] conn(0x7fd2c80bc420 msgr2=0x7fd2c80be8c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.361+0000 7fd2df0a1700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723] conn(0x7fd2c80bc420 0x7fd2c80be8c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.361+0000 7fd2df0a1700 1 -- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd2c80b2990 msgr2=0x7fd2c80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.361+0000 7fd2df0a1700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd2c80b2990 0x7fd2c80b4e50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd2d819aa50 tx=0x7fd2cc00b2c0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.361+0000 7fd2df0a1700 1 -- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d8104100 msgr2=0x7fd2d819fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.361+0000 7fd2df0a1700 1 --2- 172.21.15.67:0/509454080 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd2d8104100 0x7fd2d819fc90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400b580 tx=0x7fd2c40047b0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.361+0000 7fd2dd63e700 1 -- 172.21.15.67:0/509454080 reap_dead start 2024-05-03T17:21:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.361+0000 7fd2df0a1700 1 -- 172.21.15.67:0/509454080 shutdown_connections 2024-05-03T17:21:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.361+0000 7fd2df0a1700 1 -- 172.21.15.67:0/509454080 >> 172.21.15.67:0/509454080 conn(0x7fd2d807a600 msgr2=0x7fd2d8101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.361+0000 7fd2df0a1700 1 -- 172.21.15.67:0/509454080 shutdown_connections 2024-05-03T17:21:58.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:58.361+0000 7fd2df0a1700 1 -- 172.21.15.67:0/509454080 wait complete. 2024-05-03T17:21:58.890 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.966+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/162718949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1806bb70 msgr2=0x7f9d1806bff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.966+0000 7f9d1e0a2700 1 --2- 172.21.15.67:0/162718949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1806bb70 0x7f9d1806bff0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009990 tx=0x7f9d0c009ca0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.966+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/162718949 shutdown_connections 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.966+0000 7f9d1e0a2700 1 --2- 172.21.15.67:0/162718949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1806c530 0x7f9d18110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.966+0000 7f9d1e0a2700 1 --2- 172.21.15.67:0/162718949 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d18109530 0x7f9d18109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.966+0000 7f9d1e0a2700 1 --2- 172.21.15.67:0/162718949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1806bb70 0x7f9d1806bff0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.966+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/162718949 >> 172.21.15.67:0/162718949 conn(0x7f9d18079350 msgr2=0x7f9d18079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.966+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/162718949 shutdown_connections 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.966+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/162718949 wait complete. 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.967+0000 7f9d1e0a2700 1 Processor -- start 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.967+0000 7f9d1e0a2700 1 -- start start 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.967+0000 7f9d1e0a2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1806bb70 0x7f9d1819f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.967+0000 7f9d1e0a2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1806c530 0x7f9d1819fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.891 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.967+0000 7f9d1e0a2700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d18109530 0x7f9d18199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.967+0000 7f9d1e0a2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9d18113660 con 0x7f9d1806c530 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.967+0000 7f9d1e0a2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9d181134e0 con 0x7f9d1806bb70 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.967+0000 7f9d1e0a2700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9d181137e0 con 0x7f9d18109530 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.967+0000 7f9d16ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1806c530 0x7f9d1819fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.967+0000 7f9d16ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1806c530 0x7f9d1819fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44004/0 (socket says 172.21.15.67:44004) 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.967+0000 7f9d16ffd700 1 -- 172.21.15.67:0/709376044 learned_addr learned my addr 172.21.15.67:0/709376044 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.967+0000 7f9d17fff700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d18109530 0x7f9d18199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.968+0000 7f9d177fe700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1806bb70 0x7f9d1819f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.968+0000 7f9d16ffd700 1 -- 172.21.15.67:0/709376044 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d18109530 msgr2=0x7f9d18199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.968+0000 7f9d16ffd700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9d18109530 0x7f9d18199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.968+0000 7f9d16ffd700 1 -- 172.21.15.67:0/709376044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1806bb70 msgr2=0x7f9d1819f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.968+0000 7f9d16ffd700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9d1806bb70 0x7f9d1819f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.968+0000 7f9d16ffd700 1 -- 172.21.15.67:0/709376044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1819a0e0 con 0x7f9d1806c530 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.968+0000 7f9d16ffd700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1806c530 0x7f9d1819fd00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c000c00 tx=0x7f9d0c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:58.892 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.968+0000 7f9d14ff9700 1 -- 172.21.15.67:0/709376044 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c027070 con 0x7f9d1806c530 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.968+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/709376044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d1819a370 con 0x7f9d1806c530 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.968+0000 7f9d14ff9700 1 -- 172.21.15.67:0/709376044 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d0c0040e0 con 0x7f9d1806c530 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.968+0000 7f9d14ff9700 1 -- 172.21.15.67:0/709376044 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c00f460 con 0x7f9d1806c530 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.968+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/709376044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d181a6690 con 0x7f9d1806c530 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.969+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/709376044 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9d18063bb0 con 0x7f9d1806c530 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.970+0000 7f9d14ff9700 1 -- 172.21.15.67:0/709376044 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9d0c02a070 con 0x7f9d1806c530 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.971+0000 7f9d14ff9700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9d000b2910 0x7f9d000b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.971+0000 7f9d177fe700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9d000b2910 0x7f9d000b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.971+0000 7f9d14ff9700 1 -- 172.21.15.67:0/709376044 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f9d0c0e3a80 con 0x7f9d1806c530 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.971+0000 7f9d14ff9700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149] conn(0x7f9d000bc3a0 0x7f9d000be840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.971+0000 7f9d14ff9700 1 -- 172.21.15.67:0/709376044 --> [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d000bef10 con 0x7f9d000bc3a0 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.971+0000 7f9d14ff9700 1 -- 172.21.15.67:0/709376044 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f9d0c0e3ed0 con 0x7f9d1806c530 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.972+0000 7f9d177fe700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9d000b2910 0x7f9d000b4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9d08005fd0 tx=0x7f9d08005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.972+0000 7f9d17fff700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149] conn(0x7f9d000bc3a0 0x7f9d000be840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:21:58.893 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.972+0000 7f9d17fff700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149] conn(0x7f9d000bc3a0 0x7f9d000be840 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:21:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.973+0000 7f9d14ff9700 1 -- 172.21.15.67:0/709376044 <== osd.11 v2:172.21.15.82:6824/4082656149 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9d000bef10 con 0x7f9d000bc3a0 2024-05-03T17:21:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.990+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/709376044 --> [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9d18063fb0 con 0x7f9d000bc3a0 2024-05-03T17:21:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.990+0000 7f9d14ff9700 1 -- 172.21.15.67:0/709376044 <== osd.11 v2:172.21.15.82:6824/4082656149 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f9d18063fb0 con 0x7f9d000bc3a0 2024-05-03T17:21:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.990+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/709376044 >> [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149] conn(0x7f9d000bc3a0 msgr2=0x7f9d000be840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.990+0000 7f9d1e0a2700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149] conn(0x7f9d000bc3a0 0x7f9d000be840 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.990+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/709376044 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9d000b2910 msgr2=0x7f9d000b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.990+0000 7f9d1e0a2700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9d000b2910 0x7f9d000b4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9d08005fd0 tx=0x7f9d08005d90 comp rx=0 tx=0).stop 2024-05-03T17:21:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.990+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/709376044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1806c530 msgr2=0x7f9d1819fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:21:58.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.990+0000 7f9d1e0a2700 1 --2- 172.21.15.67:0/709376044 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9d1806c530 0x7f9d1819fd00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c000c00 tx=0x7f9d0c0047b0 comp rx=0 tx=0).stop 2024-05-03T17:21:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.991+0000 7f9d17fff700 1 -- 172.21.15.67:0/709376044 reap_dead start 2024-05-03T17:21:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.991+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/709376044 shutdown_connections 2024-05-03T17:21:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.991+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/709376044 >> 172.21.15.67:0/709376044 conn(0x7f9d18079350 msgr2=0x7f9d18101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:21:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.991+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/709376044 shutdown_connections 2024-05-03T17:21:58.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:21:57.991+0000 7f9d1e0a2700 1 -- 172.21.15.67:0/709376044 wait complete. 2024-05-03T17:22:00.253 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:00 smithi067 ceph-mon[29731]: pgmap v337: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:22:00.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:00 smithi082 ceph-mon[33535]: pgmap v337: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:22:00.290 INFO:teuthology.orchestra.run.smithi067.stdout:111669149769 2024-05-03T17:22:00.291 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-03T17:22:00.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:00 smithi070 ceph-mon[33250]: pgmap v337: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:22:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.989+0000 7f2a674d2700 1 -- 172.21.15.67:0/1632356108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a60109680 msgr2=0x7f2a60109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.989+0000 7f2a674d2700 1 --2- 172.21.15.67:0/1632356108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a60109680 0x7f2a60109a60 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c0098f0 tx=0x7f2a5c009c00 comp rx=0 tx=0).stop 2024-05-03T17:22:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.989+0000 7f2a674d2700 1 -- 172.21.15.67:0/1632356108 shutdown_connections 2024-05-03T17:22:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.989+0000 7f2a674d2700 1 --2- 172.21.15.67:0/1632356108 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a60102890 0x7f2a60110390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:01.009 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.989+0000 7f2a674d2700 1 --2- 172.21.15.67:0/1632356108 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a60109680 0x7f2a60109a60 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.989+0000 7f2a674d2700 1 --2- 172.21.15.67:0/1632356108 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a60101ed0 0x7f2a60102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.989+0000 7f2a674d2700 1 -- 172.21.15.67:0/1632356108 >> 172.21.15.67:0/1632356108 conn(0x7f2a60077610 msgr2=0x7f2a60077a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.989+0000 7f2a674d2700 1 -- 172.21.15.67:0/1632356108 shutdown_connections 2024-05-03T17:22:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.989+0000 7f2a674d2700 1 -- 172.21.15.67:0/1632356108 wait complete. 2024-05-03T17:22:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a674d2700 1 Processor -- start 2024-05-03T17:22:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a674d2700 1 -- start start 2024-05-03T17:22:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a674d2700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a60101ed0 0x7f2a6019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a674d2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a60102890 0x7f2a6019fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a674d2700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a60109680 0x7f2a601999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a674d2700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2a60113650 con 0x7f2a60102890 2024-05-03T17:22:01.010 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a674d2700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2a601134d0 con 0x7f2a60101ed0 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a674d2700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2a601137d0 con 0x7f2a60109680 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a64a6d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a60102890 0x7f2a6019fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a64a6d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a60102890 0x7f2a6019fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44064/0 (socket says 172.21.15.67:44064) 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a64a6d700 1 -- 172.21.15.67:0/834809634 learned_addr learned my addr 172.21.15.67:0/834809634 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a6526e700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a60101ed0 0x7f2a6019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a65a6f700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a60109680 0x7f2a601999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a64a6d700 1 -- 172.21.15.67:0/834809634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a60109680 msgr2=0x7f2a601999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a64a6d700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a60109680 0x7f2a601999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a64a6d700 1 -- 172.21.15.67:0/834809634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a60101ed0 msgr2=0x7f2a6019f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a64a6d700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a60101ed0 0x7f2a6019f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a64a6d700 1 -- 172.21.15.67:0/834809634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a6019a150 con 0x7f2a60102890 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a6526e700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2a60101ed0 0x7f2a6019f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:22:01.011 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.990+0000 7f2a65a6f700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2a60109680 0x7f2a601999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.991+0000 7f2a64a6d700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a60102890 0x7f2a6019fd70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000e9f0 tx=0x7f2a5000edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.991+0000 7f2a567fc700 1 -- 172.21.15.67:0/834809634 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5000cbd0 con 0x7f2a60102890 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.991+0000 7f2a567fc700 1 -- 172.21.15.67:0/834809634 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a5000cd70 con 0x7f2a60102890 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.991+0000 7f2a674d2700 1 -- 172.21.15.67:0/834809634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a6019a440 con 0x7f2a60102890 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.991+0000 7f2a567fc700 1 -- 172.21.15.67:0/834809634 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50010430 con 0x7f2a60102890 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.991+0000 7f2a674d2700 1 -- 172.21.15.67:0/834809634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a601a6700 con 0x7f2a60102890 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.992+0000 7f2a674d2700 1 -- 172.21.15.67:0/834809634 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2a60063bf0 con 0x7f2a60102890 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.993+0000 7f2a567fc700 1 -- 172.21.15.67:0/834809634 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f2a50004750 con 0x7f2a60102890 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.994+0000 7f2a567fc700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f2a4c0b2910 0x7f2a4c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.994+0000 7f2a6526e700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f2a4c0b2910 0x7f2a4c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.994+0000 7f2a567fc700 1 -- 172.21.15.67:0/834809634 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f2a500e2b50 con 0x7f2a60102890 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.994+0000 7f2a567fc700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176] conn(0x7f2a4c0bc4d0 0x7f2a4c0be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:01.012 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.994+0000 7f2a567fc700 1 -- 172.21.15.67:0/834809634 --> [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a4c0bf040 con 0x7f2a4c0bc4d0 2024-05-03T17:22:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.994+0000 7f2a567fc700 1 -- 172.21.15.67:0/834809634 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f2a500e2fa0 con 0x7f2a60102890 2024-05-03T17:22:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.994+0000 7f2a6526e700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f2a4c0b2910 0x7f2a4c0b4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009230 tx=0x7f2a5c005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.994+0000 7f2a65a6f700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176] conn(0x7f2a4c0bc4d0 0x7f2a4c0be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.995+0000 7f2a65a6f700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176] conn(0x7f2a4c0bc4d0 0x7f2a4c0be970 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:01.013 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:00.996+0000 7f2a567fc700 1 -- 172.21.15.67:0/834809634 <== osd.1 v2:172.21.15.70:6800/1006339176 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2a4c0bf040 con 0x7f2a4c0bc4d0 2024-05-03T17:22:01.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.013+0000 7f2a674d2700 1 -- 172.21.15.67:0/834809634 --> [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2a60064070 con 0x7f2a4c0bc4d0 2024-05-03T17:22:01.014 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.013+0000 7f2a567fc700 1 -- 172.21.15.67:0/834809634 <== osd.1 v2:172.21.15.70:6800/1006339176 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f2a60064070 con 0x7f2a4c0bc4d0 2024-05-03T17:22:01.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.013+0000 7f2a674d2700 1 -- 172.21.15.67:0/834809634 >> [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176] conn(0x7f2a4c0bc4d0 msgr2=0x7f2a4c0be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:01.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.013+0000 7f2a674d2700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176] conn(0x7f2a4c0bc4d0 0x7f2a4c0be970 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:01.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.014+0000 7f2a674d2700 1 -- 172.21.15.67:0/834809634 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f2a4c0b2910 msgr2=0x7f2a4c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:01.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.014+0000 7f2a674d2700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f2a4c0b2910 0x7f2a4c0b4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009230 tx=0x7f2a5c005dc0 comp rx=0 tx=0).stop 2024-05-03T17:22:01.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.014+0000 7f2a674d2700 1 -- 172.21.15.67:0/834809634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a60102890 msgr2=0x7f2a6019fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:01.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.014+0000 7f2a674d2700 1 --2- 172.21.15.67:0/834809634 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a60102890 0x7f2a6019fd70 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f2a5000e9f0 tx=0x7f2a5000edb0 comp rx=0 tx=0).stop 2024-05-03T17:22:01.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.014+0000 7f2a65a6f700 1 -- 172.21.15.67:0/834809634 reap_dead start 2024-05-03T17:22:01.015 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.014+0000 7f2a674d2700 1 -- 172.21.15.67:0/834809634 shutdown_connections 2024-05-03T17:22:01.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.014+0000 7f2a674d2700 1 -- 172.21.15.67:0/834809634 >> 172.21.15.67:0/834809634 conn(0x7f2a60077610 msgr2=0x7f2a60100b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:01.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.014+0000 7f2a674d2700 1 -- 172.21.15.67:0/834809634 shutdown_connections 2024-05-03T17:22:01.016 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:01.014+0000 7f2a674d2700 1 -- 172.21.15.67:0/834809634 wait complete. 2024-05-03T17:22:01.233 INFO:teuthology.orchestra.run.smithi067.stdout:34359738455 2024-05-03T17:22:01.233 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-03T17:22:02.211 INFO:teuthology.orchestra.run.smithi067.stdout:231928234033 2024-05-03T17:22:02.212 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-03T17:22:02.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:02 smithi082 ceph-mon[33535]: pgmap v338: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:22:02.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:02 smithi070 ceph-mon[33250]: pgmap v338: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:22:02.312 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:02 smithi067 ceph-mon[29731]: pgmap v338: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:22:02.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.420+0000 7f2769a21700 1 -- 172.21.15.67:0/4042884331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2764104130 msgr2=0x7f2764108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:02.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.420+0000 7f2769a21700 1 --2- 172.21.15.67:0/4042884331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2764104130 0x7f2764108140 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f27540099c0 tx=0x7f2754009cd0 comp rx=0 tx=0).stop 2024-05-03T17:22:02.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.420+0000 7f2769a21700 1 -- 172.21.15.67:0/4042884331 shutdown_connections 2024-05-03T17:22:02.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.420+0000 7f2769a21700 1 --2- 172.21.15.67:0/4042884331 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2764108870 0x7f27641103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:02.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.420+0000 7f2769a21700 1 --2- 172.21.15.67:0/4042884331 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2764103780 0x7f2764103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:02.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.420+0000 7f2769a21700 1 --2- 172.21.15.67:0/4042884331 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2764104130 0x7f2764108140 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:02.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.420+0000 7f2769a21700 1 -- 172.21.15.67:0/4042884331 >> 172.21.15.67:0/4042884331 conn(0x7f276407a610 msgr2=0x7f276407aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:02.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.420+0000 7f2769a21700 1 -- 172.21.15.67:0/4042884331 shutdown_connections 2024-05-03T17:22:02.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.420+0000 7f2769a21700 1 -- 172.21.15.67:0/4042884331 wait complete. 2024-05-03T17:22:02.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f2769a21700 1 Processor -- start 2024-05-03T17:22:02.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f2769a21700 1 -- start start 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f2769a21700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2764103780 0x7f276419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f2769a21700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2764104130 0x7f276419fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f2769a21700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2764108870 0x7f27641998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f2769a21700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2764113650 con 0x7f2764108870 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f2769a21700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f27641134d0 con 0x7f2764103780 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f2769a21700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f27641137d0 con 0x7f2764104130 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f27637fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2764108870 0x7f27641998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f27637fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2764108870 0x7f27641998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44090/0 (socket says 172.21.15.67:44090) 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f27637fe700 1 -- 172.21.15.67:0/2648425241 learned_addr learned my addr 172.21.15.67:0/2648425241 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f27627fc700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2764104130 0x7f276419fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.421+0000 7f2762ffd700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2764103780 0x7f276419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.422+0000 7f27637fe700 1 -- 172.21.15.67:0/2648425241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2764104130 msgr2=0x7f276419fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:02.430 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.422+0000 7f27637fe700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2764104130 0x7f276419fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.422+0000 7f27637fe700 1 -- 172.21.15.67:0/2648425241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2764103780 msgr2=0x7f276419f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.422+0000 7f27637fe700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2764103780 0x7f276419f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.422+0000 7f27637fe700 1 -- 172.21.15.67:0/2648425241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f276419a050 con 0x7f2764108870 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.422+0000 7f27637fe700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2764108870 0x7f27641998f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f275c007d70 tx=0x7f275c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.422+0000 7f2768a1f700 1 -- 172.21.15.67:0/2648425241 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c016020 con 0x7f2764108870 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.422+0000 7f2768a1f700 1 -- 172.21.15.67:0/2648425241 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f275c00ac90 con 0x7f2764108870 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.422+0000 7f2769a21700 1 -- 172.21.15.67:0/2648425241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f276419a340 con 0x7f2764108870 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.422+0000 7f2768a1f700 1 -- 172.21.15.67:0/2648425241 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f275c0115e0 con 0x7f2764108870 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.422+0000 7f2769a21700 1 -- 172.21.15.67:0/2648425241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27641a65b0 con 0x7f2764108870 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.423+0000 7f2769a21700 1 -- 172.21.15.67:0/2648425241 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2764063c30 con 0x7f2764108870 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.424+0000 7f2768a1f700 1 -- 172.21.15.67:0/2648425241 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f275c019020 con 0x7f2764108870 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.425+0000 7f2768a1f700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f274c0b2910 0x7f274c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.425+0000 7f2762ffd700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f274c0b2910 0x7f274c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.425+0000 7f2768a1f700 1 -- 172.21.15.67:0/2648425241 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f275c0e26e0 con 0x7f2764108870 2024-05-03T17:22:02.431 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.425+0000 7f2768a1f700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161] conn(0x7f274c0bc4d0 0x7f274c0be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:02.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.425+0000 7f2768a1f700 1 -- 172.21.15.67:0/2648425241 --> [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f274c0bf040 con 0x7f274c0bc4d0 2024-05-03T17:22:02.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.425+0000 7f2768a1f700 1 -- 172.21.15.67:0/2648425241 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f275c0e2b30 con 0x7f2764108870 2024-05-03T17:22:02.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.425+0000 7f27627fc700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161] conn(0x7f274c0bc4d0 0x7f274c0be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:02.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.426+0000 7f2762ffd700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f274c0b2910 0x7f274c0b4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2764108320 tx=0x7f275800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:02.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.426+0000 7f27627fc700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161] conn(0x7f274c0bc4d0 0x7f274c0be970 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:02.432 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.426+0000 7f2768a1f700 1 -- 172.21.15.67:0/2648425241 <== osd.6 v2:172.21.15.67:6818/4033769161 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f274c0bf040 con 0x7f274c0bc4d0 2024-05-03T17:22:02.445 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.445+0000 7f2769a21700 1 -- 172.21.15.67:0/2648425241 --> [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f27640640b0 con 0x7f274c0bc4d0 2024-05-03T17:22:02.446 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.445+0000 7f2768a1f700 1 -- 172.21.15.67:0/2648425241 <== osd.6 v2:172.21.15.67:6818/4033769161 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f27640640b0 con 0x7f274c0bc4d0 2024-05-03T17:22:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.446+0000 7f2769a21700 1 -- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161] conn(0x7f274c0bc4d0 msgr2=0x7f274c0be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.446+0000 7f2769a21700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161] conn(0x7f274c0bc4d0 0x7f274c0be970 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.446+0000 7f2769a21700 1 -- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f274c0b2910 msgr2=0x7f274c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.446+0000 7f2769a21700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f274c0b2910 0x7f274c0b4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2764108320 tx=0x7f275800b3c0 comp rx=0 tx=0).stop 2024-05-03T17:22:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.446+0000 7f2769a21700 1 -- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2764108870 msgr2=0x7f27641998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.446+0000 7f2769a21700 1 --2- 172.21.15.67:0/2648425241 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2764108870 0x7f27641998f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f275c007d70 tx=0x7f275c00a340 comp rx=0 tx=0).stop 2024-05-03T17:22:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.446+0000 7f27637fe700 1 -- 172.21.15.67:0/2648425241 reap_dead start 2024-05-03T17:22:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.446+0000 7f2769a21700 1 -- 172.21.15.67:0/2648425241 shutdown_connections 2024-05-03T17:22:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.446+0000 7f2769a21700 1 -- 172.21.15.67:0/2648425241 >> 172.21.15.67:0/2648425241 conn(0x7f276407a610 msgr2=0x7f27641017f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.446+0000 7f2769a21700 1 -- 172.21.15.67:0/2648425241 shutdown_connections 2024-05-03T17:22:02.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:02.446+0000 7f2769a21700 1 -- 172.21.15.67:0/2648425241 wait complete. 2024-05-03T17:22:03.643 INFO:teuthology.orchestra.run.smithi067.stdout:55834574933 2024-05-03T17:22:03.643 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-03T17:22:04.090 INFO:teuthology.orchestra.run.smithi067.stdout:304942678049 2024-05-03T17:22:04.091 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-03T17:22:04.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:04 smithi082 ceph-mon[33535]: pgmap v339: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:22:04.294 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:04 smithi067 ceph-mon[29731]: pgmap v339: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:22:04.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:04 smithi070 ceph-mon[33250]: pgmap v339: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:22:06.078 INFO:teuthology.orchestra.run.smithi067.stdout:184683593788 2024-05-03T17:22:06.079 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-03T17:22:06.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:06 smithi082 ceph-mon[33535]: pgmap v340: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:22:06.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:06 smithi070 ceph-mon[33250]: pgmap v340: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:22:06.373 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:06 smithi067 ceph-mon[29731]: pgmap v340: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:22:07.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.134+0000 7f088594c700 1 -- 172.21.15.67:0/3055280251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0880108440 msgr2=0x7f0880108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:07.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.134+0000 7f088594c700 1 --2- 172.21.15.67:0/3055280251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0880108440 0x7f0880108820 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f0870009990 tx=0x7f0870009ca0 comp rx=0 tx=0).stop 2024-05-03T17:22:07.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.134+0000 7f088594c700 1 -- 172.21.15.67:0/3055280251 shutdown_connections 2024-05-03T17:22:07.135 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.134+0000 7f088594c700 1 --2- 172.21.15.67:0/3055280251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0880102d90 0x7f088010f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.134+0000 7f088594c700 1 --2- 172.21.15.67:0/3055280251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0880108440 0x7f0880108820 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.134+0000 7f088594c700 1 --2- 172.21.15.67:0/3055280251 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f08801023d0 0x7f0880102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.134+0000 7f088594c700 1 -- 172.21.15.67:0/3055280251 >> 172.21.15.67:0/3055280251 conn(0x7f08800793e0 msgr2=0x7f08800797f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.134+0000 7f088594c700 1 -- 172.21.15.67:0/3055280251 shutdown_connections 2024-05-03T17:22:07.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.134+0000 7f088594c700 1 -- 172.21.15.67:0/3055280251 wait complete. 2024-05-03T17:22:07.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.135+0000 7f088594c700 1 Processor -- start 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.135+0000 7f088594c700 1 -- start start 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f088594c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08801023d0 0x7f088019f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f088594c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0880102d90 0x7f088019fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f088594c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0880108440 0x7f08801999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f088594c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0880112590 con 0x7f08801023d0 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f088594c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0880112410 con 0x7f0880108440 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f088594c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0880112710 con 0x7f0880102d90 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08801023d0 0x7f088019f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08801023d0 0x7f088019f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52452/0 (socket says 172.21.15.67:52452) 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087effd700 1 -- 172.21.15.67:0/2547261405 learned_addr learned my addr 172.21.15.67:0/2547261405 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087e7fc700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0880102d90 0x7f088019fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087f7fe700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0880108440 0x7f08801999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:07.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087effd700 1 -- 172.21.15.67:0/2547261405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0880102d90 msgr2=0x7f088019fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087effd700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0880102d90 0x7f088019fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087effd700 1 -- 172.21.15.67:0/2547261405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0880108440 msgr2=0x7f08801999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087effd700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0880108440 0x7f08801999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087effd700 1 -- 172.21.15.67:0/2547261405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f088019a130 con 0x7f08801023d0 2024-05-03T17:22:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087f7fe700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0880108440 0x7f08801999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:22:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087e7fc700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0880102d90 0x7f088019fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:22:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.136+0000 7f087effd700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08801023d0 0x7f088019f810 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f087000b2e0 tx=0x7f0870004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.137+0000 7f088494a700 1 -- 172.21.15.67:0/2547261405 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0870026070 con 0x7f08801023d0 2024-05-03T17:22:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.137+0000 7f088594c700 1 -- 172.21.15.67:0/2547261405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f088019a3c0 con 0x7f08801023d0 2024-05-03T17:22:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.137+0000 7f088494a700 1 -- 172.21.15.67:0/2547261405 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0870021410 con 0x7f08801023d0 2024-05-03T17:22:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.137+0000 7f088594c700 1 -- 172.21.15.67:0/2547261405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08801a66e0 con 0x7f08801023d0 2024-05-03T17:22:07.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.137+0000 7f088494a700 1 -- 172.21.15.67:0/2547261405 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0870017650 con 0x7f08801023d0 2024-05-03T17:22:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.138+0000 7f088594c700 1 -- 172.21.15.67:0/2547261405 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f086c001030 con 0x7f08801023d0 2024-05-03T17:22:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.139+0000 7f088494a700 1 -- 172.21.15.67:0/2547261405 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0870029070 con 0x7f08801023d0 2024-05-03T17:22:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.140+0000 7f088494a700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f08680b2990 0x7f08680b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.140+0000 7f087e7fc700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f08680b2990 0x7f08680b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.140+0000 7f088494a700 1 -- 172.21.15.67:0/2547261405 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f08700e3f40 con 0x7f08801023d0 2024-05-03T17:22:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.140+0000 7f088494a700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828] conn(0x7f08680bc550 0x7f08680be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.140+0000 7f088494a700 1 -- 172.21.15.67:0/2547261405 --> [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08680bf0c0 con 0x7f08680bc550 2024-05-03T17:22:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.140+0000 7f088494a700 1 -- 172.21.15.67:0/2547261405 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f08700e4390 con 0x7f08801023d0 2024-05-03T17:22:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.140+0000 7f087e7fc700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f08680b2990 0x7f08680b4e50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0880079950 tx=0x7f0874009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:07.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.141+0000 7f087f7fe700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828] conn(0x7f08680bc550 0x7f08680be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:07.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.141+0000 7f087f7fe700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828] conn(0x7f08680bc550 0x7f08680be9f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:07.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.142+0000 7f088494a700 1 -- 172.21.15.67:0/2547261405 <== osd.4 v2:172.21.15.70:6808/3526300828 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08680bf0c0 con 0x7f08680bc550 2024-05-03T17:22:07.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.159+0000 7f088594c700 1 -- 172.21.15.67:0/2547261405 --> [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f086c002e60 con 0x7f08680bc550 2024-05-03T17:22:07.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.159+0000 7f088494a700 1 -- 172.21.15.67:0/2547261405 <== osd.4 v2:172.21.15.70:6808/3526300828 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f086c002e60 con 0x7f08680bc550 2024-05-03T17:22:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.159+0000 7f088594c700 1 -- 172.21.15.67:0/2547261405 >> [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828] conn(0x7f08680bc550 msgr2=0x7f08680be9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.159+0000 7f088594c700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828] conn(0x7f08680bc550 0x7f08680be9f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.159+0000 7f088594c700 1 -- 172.21.15.67:0/2547261405 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f08680b2990 msgr2=0x7f08680b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.159+0000 7f088594c700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f08680b2990 0x7f08680b4e50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0880079950 tx=0x7f0874009260 comp rx=0 tx=0).stop 2024-05-03T17:22:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.160+0000 7f088594c700 1 -- 172.21.15.67:0/2547261405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08801023d0 msgr2=0x7f088019f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:07.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.160+0000 7f088594c700 1 --2- 172.21.15.67:0/2547261405 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f08801023d0 0x7f088019f810 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f087000b2e0 tx=0x7f0870004790 comp rx=0 tx=0).stop 2024-05-03T17:22:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.160+0000 7f087f7fe700 1 -- 172.21.15.67:0/2547261405 reap_dead start 2024-05-03T17:22:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.160+0000 7f088594c700 1 -- 172.21.15.67:0/2547261405 shutdown_connections 2024-05-03T17:22:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.160+0000 7f088594c700 1 -- 172.21.15.67:0/2547261405 >> 172.21.15.67:0/2547261405 conn(0x7f08800793e0 msgr2=0x7f0880101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.160+0000 7f088594c700 1 -- 172.21.15.67:0/2547261405 shutdown_connections 2024-05-03T17:22:07.161 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:07.160+0000 7f088594c700 1 -- 172.21.15.67:0/2547261405 wait complete. 2024-05-03T17:22:07.906 INFO:teuthology.orchestra.run.smithi067.stdout:137438953543 2024-05-03T17:22:07.907 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-03T17:22:08.131 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:08 smithi067 ceph-mon[29731]: pgmap v341: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:08.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:08 smithi070 ceph-mon[33250]: pgmap v341: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:08.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:08 smithi082 ceph-mon[33535]: pgmap v341: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:08.878 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.877+0000 7f5976d29700 1 -- 172.21.15.67:0/3165472882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5970103750 msgr2=0x7f5970103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.877+0000 7f5976d29700 1 --2- 172.21.15.67:0/3165472882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5970103750 0x7f5970103b30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f59600099c0 tx=0x7f5960009cd0 comp rx=0 tx=0).stop 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.877+0000 7f5976d29700 1 -- 172.21.15.67:0/3165472882 shutdown_connections 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.877+0000 7f5976d29700 1 --2- 172.21.15.67:0/3165472882 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5970108850 0x7f59701103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.877+0000 7f5976d29700 1 --2- 172.21.15.67:0/3165472882 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5970103750 0x7f5970103b30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.877+0000 7f5976d29700 1 --2- 172.21.15.67:0/3165472882 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5970104100 0x7f5970108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.877+0000 7f5976d29700 1 -- 172.21.15.67:0/3165472882 >> 172.21.15.67:0/3165472882 conn(0x7f597007a600 msgr2=0x7f597007aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.877+0000 7f5976d29700 1 -- 172.21.15.67:0/3165472882 shutdown_connections 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.877+0000 7f5976d29700 1 -- 172.21.15.67:0/3165472882 wait complete. 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.877+0000 7f5976d29700 1 Processor -- start 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.877+0000 7f5976d29700 1 -- start start 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f5976d29700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5970103750 0x7f597019f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f5976d29700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5970104100 0x7f597019fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f5976d29700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5970108850 0x7f59701998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:08.879 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f5976d29700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f5970113620 con 0x7f5970104100 2024-05-03T17:22:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f5976d29700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f59701134a0 con 0x7f5970108850 2024-05-03T17:22:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f5976d29700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f59701137a0 con 0x7f5970103750 2024-05-03T17:22:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f596ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5970104100 0x7f597019fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f596ffff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5970104100 0x7f597019fc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52460/0 (socket says 172.21.15.67:52460) 2024-05-03T17:22:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f596ffff700 1 -- 172.21.15.67:0/3369768134 learned_addr learned my addr 172.21.15.67:0/3369768134 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f59752c6700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5970108850 0x7f59701998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f5974ac5700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5970103750 0x7f597019f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f596ffff700 1 -- 172.21.15.67:0/3369768134 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5970103750 msgr2=0x7f597019f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f596ffff700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5970103750 0x7f597019f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f596ffff700 1 -- 172.21.15.67:0/3369768134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5970108850 msgr2=0x7f59701998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f596ffff700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5970108850 0x7f59701998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:08.880 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f596ffff700 1 -- 172.21.15.67:0/3369768134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597019a000 con 0x7f5970104100 2024-05-03T17:22:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.878+0000 7f596ffff700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5970104100 0x7f597019fc20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f596400d950 tx=0x7f596400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.879+0000 7f596dffb700 1 -- 172.21.15.67:0/3369768134 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5964016070 con 0x7f5970104100 2024-05-03T17:22:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.879+0000 7f596dffb700 1 -- 172.21.15.67:0/3369768134 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f596400fd40 con 0x7f5970104100 2024-05-03T17:22:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.879+0000 7f5976d29700 1 -- 172.21.15.67:0/3369768134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f597019a2f0 con 0x7f5970104100 2024-05-03T17:22:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.879+0000 7f596dffb700 1 -- 172.21.15.67:0/3369768134 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f596400e3b0 con 0x7f5970104100 2024-05-03T17:22:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.879+0000 7f5976d29700 1 -- 172.21.15.67:0/3369768134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59701a65b0 con 0x7f5970104100 2024-05-03T17:22:08.881 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.880+0000 7f5976d29700 1 -- 172.21.15.67:0/3369768134 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5970063c30 con 0x7f5970104100 2024-05-03T17:22:08.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.881+0000 7f596dffb700 1 -- 172.21.15.67:0/3369768134 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f5964019070 con 0x7f5970104100 2024-05-03T17:22:08.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.882+0000 7f596dffb700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f59580b2990 0x7f59580b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:08.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.882+0000 7f5974ac5700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f59580b2990 0x7f59580b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:08.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.882+0000 7f596dffb700 1 -- 172.21.15.67:0/3369768134 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f59640e2310 con 0x7f5970104100 2024-05-03T17:22:08.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.882+0000 7f596dffb700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629] conn(0x7f59580bc550 0x7f59580be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:08.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.882+0000 7f596dffb700 1 -- 172.21.15.67:0/3369768134 --> [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f59580bf0c0 con 0x7f59580bc550 2024-05-03T17:22:08.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.882+0000 7f596dffb700 1 -- 172.21.15.67:0/3369768134 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f59640e2760 con 0x7f5970104100 2024-05-03T17:22:08.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.882+0000 7f5974ac5700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f59580b2990 0x7f59580b4e50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f597019a9e0 tx=0x7f5960005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:08.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.882+0000 7f59752c6700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629] conn(0x7f59580bc550 0x7f59580be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:08.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.883+0000 7f59752c6700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629] conn(0x7f59580bc550 0x7f59580be9f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:08.884 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.884+0000 7f596dffb700 1 -- 172.21.15.67:0/3369768134 <== osd.10 v2:172.21.15.70:6824/2897583629 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f59580bf0c0 con 0x7f59580bc550 2024-05-03T17:22:08.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.900+0000 7f5976d29700 1 -- 172.21.15.67:0/3369768134 --> [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5970064030 con 0x7f59580bc550 2024-05-03T17:22:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.900+0000 7f596dffb700 1 -- 172.21.15.67:0/3369768134 <== osd.10 v2:172.21.15.70:6824/2897583629 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5970064030 con 0x7f59580bc550 2024-05-03T17:22:08.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.901+0000 7f5976d29700 1 -- 172.21.15.67:0/3369768134 >> [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629] conn(0x7f59580bc550 msgr2=0x7f59580be9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.901+0000 7f5976d29700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629] conn(0x7f59580bc550 0x7f59580be9f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.901+0000 7f5976d29700 1 -- 172.21.15.67:0/3369768134 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f59580b2990 msgr2=0x7f59580b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.901+0000 7f5976d29700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f59580b2990 0x7f59580b4e50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f597019a9e0 tx=0x7f5960005c40 comp rx=0 tx=0).stop 2024-05-03T17:22:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.901+0000 7f5976d29700 1 -- 172.21.15.67:0/3369768134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5970104100 msgr2=0x7f597019fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.901+0000 7f5976d29700 1 --2- 172.21.15.67:0/3369768134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5970104100 0x7f597019fc20 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f596400d950 tx=0x7f596400dc60 comp rx=0 tx=0).stop 2024-05-03T17:22:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.901+0000 7f59752c6700 1 -- 172.21.15.67:0/3369768134 reap_dead start 2024-05-03T17:22:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.901+0000 7f5976d29700 1 -- 172.21.15.67:0/3369768134 shutdown_connections 2024-05-03T17:22:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.901+0000 7f5976d29700 1 -- 172.21.15.67:0/3369768134 >> 172.21.15.67:0/3369768134 conn(0x7f597007a600 msgr2=0x7f5970101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.901+0000 7f5976d29700 1 -- 172.21.15.67:0/3369768134 shutdown_connections 2024-05-03T17:22:08.902 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:08.901+0000 7f5976d29700 1 -- 172.21.15.67:0/3369768134 wait complete. 2024-05-03T17:22:09.645 INFO:teuthology.orchestra.run.smithi067.stdout:279172874280 2024-05-03T17:22:09.646 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-03T17:22:09.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.841+0000 7f79349a7700 1 -- 172.21.15.67:0/3675352446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7930109510 msgr2=0x7f79301098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:09.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.841+0000 7f79349a7700 1 --2- 172.21.15.67:0/3675352446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7930109510 0x7f79301098f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7918009990 tx=0x7f7918009ca0 comp rx=0 tx=0).stop 2024-05-03T17:22:09.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.841+0000 7f79349a7700 1 -- 172.21.15.67:0/3675352446 shutdown_connections 2024-05-03T17:22:09.846 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.841+0000 7f79349a7700 1 --2- 172.21.15.67:0/3675352446 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f793006bb40 0x7f7930110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.841+0000 7f79349a7700 1 --2- 172.21.15.67:0/3675352446 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7930109510 0x7f79301098f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.841+0000 7f79349a7700 1 --2- 172.21.15.67:0/3675352446 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793006b180 0x7f793006b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.841+0000 7f79349a7700 1 -- 172.21.15.67:0/3675352446 >> 172.21.15.67:0/3675352446 conn(0x7f7930079350 msgr2=0x7f7930079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.841+0000 7f79349a7700 1 -- 172.21.15.67:0/3675352446 shutdown_connections 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.841+0000 7f79349a7700 1 -- 172.21.15.67:0/3675352446 wait complete. 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.842+0000 7f79349a7700 1 Processor -- start 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.842+0000 7f79349a7700 1 -- start start 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.842+0000 7f79349a7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793006b180 0x7f7930102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.842+0000 7f79349a7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793006bb40 0x7f79301029b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.842+0000 7f79349a7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7930109510 0x7f7930102ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.842+0000 7f79349a7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7930113690 con 0x7f793006bb40 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.842+0000 7f79349a7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7930113510 con 0x7f7930109510 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.842+0000 7f79349a7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7930113810 con 0x7f793006b180 2024-05-03T17:22:09.847 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f792dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793006bb40 0x7f79301029b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f792dd9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793006bb40 0x7f79301029b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52472/0 (socket says 172.21.15.67:52472) 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f792dd9b700 1 -- 172.21.15.67:0/2400268625 learned_addr learned my addr 172.21.15.67:0/2400268625 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f792e59c700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793006b180 0x7f7930102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f792ed9d700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7930109510 0x7f7930102ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f792dd9b700 1 -- 172.21.15.67:0/2400268625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793006b180 msgr2=0x7f7930102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f792dd9b700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f793006b180 0x7f7930102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f792dd9b700 1 -- 172.21.15.67:0/2400268625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7930109510 msgr2=0x7f7930102ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f792dd9b700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7930109510 0x7f7930102ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f792dd9b700 1 -- 172.21.15.67:0/2400268625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79301a80f0 con 0x7f793006bb40 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f792ed9d700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7930109510 0x7f7930102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f792dd9b700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793006bb40 0x7f79301029b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f792000d980 tx=0x7f792000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f79277fe700 1 -- 172.21.15.67:0/2400268625 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7920017070 con 0x7f793006bb40 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f79349a7700 1 -- 172.21.15.67:0/2400268625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79301a83e0 con 0x7f793006bb40 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f79277fe700 1 -- 172.21.15.67:0/2400268625 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7920005020 con 0x7f793006bb40 2024-05-03T17:22:09.848 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f79277fe700 1 -- 172.21.15.67:0/2400268625 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7920009b50 con 0x7f793006bb40 2024-05-03T17:22:09.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.843+0000 7f79349a7700 1 -- 172.21.15.67:0/2400268625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79301a8920 con 0x7f793006bb40 2024-05-03T17:22:09.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.844+0000 7f79349a7700 1 -- 172.21.15.67:0/2400268625 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7930063bb0 con 0x7f793006bb40 2024-05-03T17:22:09.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.846+0000 7f79277fe700 1 -- 172.21.15.67:0/2400268625 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f79200051d0 con 0x7f793006bb40 2024-05-03T17:22:09.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.846+0000 7f79277fe700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f791c0b2910 0x7f791c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:09.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.847+0000 7f79277fe700 1 -- 172.21.15.67:0/2400268625 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f79200e28b0 con 0x7f793006bb40 2024-05-03T17:22:09.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.847+0000 7f79277fe700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290] conn(0x7f791c0bc4d0 0x7f791c0be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:09.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.847+0000 7f79277fe700 1 -- 172.21.15.67:0/2400268625 --> [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f791c0bf040 con 0x7f791c0bc4d0 2024-05-03T17:22:09.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.847+0000 7f79277fe700 1 -- 172.21.15.67:0/2400268625 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f79200e2d00 con 0x7f793006bb40 2024-05-03T17:22:09.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.847+0000 7f792e59c700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f791c0b2910 0x7f791c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:09.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.848+0000 7f792ed9d700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290] conn(0x7f791c0bc4d0 0x7f791c0be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:09.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.849+0000 7f792e59c700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f791c0b2910 0x7f791c0b4dd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f791800b5c0 tx=0x7f7918005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:09.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.849+0000 7f792ed9d700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290] conn(0x7f791c0bc4d0 0x7f791c0be970 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:09.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.850+0000 7f79277fe700 1 -- 172.21.15.67:0/2400268625 <== osd.5 v2:172.21.15.82:6808/3754223290 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f791c0bf040 con 0x7f791c0bc4d0 2024-05-03T17:22:09.868 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:09.868 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:09.868 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:09.868 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:09.869 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:09.887 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.886+0000 7f79349a7700 1 -- 172.21.15.67:0/2400268625 --> [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7930064030 con 0x7f791c0bc4d0 2024-05-03T17:22:09.888 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.887+0000 7f79277fe700 1 -- 172.21.15.67:0/2400268625 <== osd.5 v2:172.21.15.82:6808/3754223290 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7930064030 con 0x7f791c0bc4d0 2024-05-03T17:22:09.894 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.894+0000 7f79349a7700 1 -- 172.21.15.67:0/2400268625 >> [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290] conn(0x7f791c0bc4d0 msgr2=0x7f791c0be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:09.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.894+0000 7f79349a7700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290] conn(0x7f791c0bc4d0 0x7f791c0be970 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:09.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.894+0000 7f79349a7700 1 -- 172.21.15.67:0/2400268625 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f791c0b2910 msgr2=0x7f791c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:09.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.894+0000 7f79349a7700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f791c0b2910 0x7f791c0b4dd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f791800b5c0 tx=0x7f7918005fd0 comp rx=0 tx=0).stop 2024-05-03T17:22:09.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.894+0000 7f79349a7700 1 -- 172.21.15.67:0/2400268625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793006bb40 msgr2=0x7f79301029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:09.895 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.894+0000 7f79349a7700 1 --2- 172.21.15.67:0/2400268625 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f793006bb40 0x7f79301029b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f792000d980 tx=0x7f792000dc90 comp rx=0 tx=0).stop 2024-05-03T17:22:09.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.895+0000 7f792ed9d700 1 -- 172.21.15.67:0/2400268625 reap_dead start 2024-05-03T17:22:09.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.895+0000 7f79349a7700 1 -- 172.21.15.67:0/2400268625 shutdown_connections 2024-05-03T17:22:09.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.895+0000 7f79349a7700 1 -- 172.21.15.67:0/2400268625 >> 172.21.15.67:0/2400268625 conn(0x7f7930079350 msgr2=0x7f79301011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:09.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.896+0000 7f79349a7700 1 -- 172.21.15.67:0/2400268625 shutdown_connections 2024-05-03T17:22:09.896 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:09.896+0000 7f79349a7700 1 -- 172.21.15.67:0/2400268625 wait complete. 2024-05-03T17:22:10.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:10 smithi067 ceph-mon[29731]: pgmap v342: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:22:10.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:10 smithi070 ceph-mon[33250]: pgmap v342: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:22:10.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:10 smithi082 ceph-mon[33535]: pgmap v342: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:22:12.238 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:12 smithi070 ceph-mon[33250]: pgmap v343: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:22:12.344 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:12 smithi067 ceph-mon[29731]: pgmap v343: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:22:12.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:12 smithi082 ceph-mon[33535]: pgmap v343: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:22:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:13.268 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.342+0000 7feba34bd700 1 -- 172.21.15.67:0/2755240772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9c06c590 msgr2=0x7feb9c110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.342+0000 7feba34bd700 1 --2- 172.21.15.67:0/2755240772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9c06c590 0x7feb9c110330 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7feb9400b0a0 tx=0x7feb9400b3b0 comp rx=0 tx=0).stop 2024-05-03T17:22:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.342+0000 7feba34bd700 1 -- 172.21.15.67:0/2755240772 shutdown_connections 2024-05-03T17:22:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.342+0000 7feba34bd700 1 --2- 172.21.15.67:0/2755240772 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9c06c590 0x7feb9c110330 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.342+0000 7feba34bd700 1 --2- 172.21.15.67:0/2755240772 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb9c109510 0x7feb9c1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.342+0000 7feba34bd700 1 --2- 172.21.15.67:0/2755240772 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feb9c06bbd0 0x7feb9c06c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.342+0000 7feba34bd700 1 -- 172.21.15.67:0/2755240772 >> 172.21.15.67:0/2755240772 conn(0x7feb9c079350 msgr2=0x7feb9c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.342+0000 7feba34bd700 1 -- 172.21.15.67:0/2755240772 shutdown_connections 2024-05-03T17:22:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.342+0000 7feba34bd700 1 -- 172.21.15.67:0/2755240772 wait complete. 2024-05-03T17:22:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.343+0000 7feba34bd700 1 Processor -- start 2024-05-03T17:22:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.343+0000 7feba34bd700 1 -- start start 2024-05-03T17:22:13.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.343+0000 7feba34bd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9c06bbd0 0x7feb9c074cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.343+0000 7feba34bd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb9c06c590 0x7feb9c06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.343+0000 7feba34bd700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feb9c109510 0x7feb9c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.343+0000 7feba34bd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feb9c113720 con 0x7feb9c06bbd0 2024-05-03T17:22:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.343+0000 7feba34bd700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feb9c1135a0 con 0x7feb9c06c590 2024-05-03T17:22:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.343+0000 7feba34bd700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7feb9c1138a0 con 0x7feb9c109510 2024-05-03T17:22:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.343+0000 7feba1259700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9c06bbd0 0x7feb9c074cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.343+0000 7feba1259700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9c06bbd0 0x7feb9c074cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:52482/0 (socket says 172.21.15.67:52482) 2024-05-03T17:22:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feba1259700 1 -- 172.21.15.67:0/1280337105 learned_addr learned my addr 172.21.15.67:0/1280337105 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feba0a58700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb9c06c590 0x7feb9c06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:13.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feba1a5a700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feb9c109510 0x7feb9c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feba1259700 1 -- 172.21.15.67:0/1280337105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feb9c109510 msgr2=0x7feb9c06e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feba1259700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feb9c109510 0x7feb9c06e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feba1259700 1 -- 172.21.15.67:0/1280337105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb9c06c590 msgr2=0x7feb9c06ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feba1259700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feb9c06c590 0x7feb9c06ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feba1259700 1 -- 172.21.15.67:0/1280337105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb9c06e930 con 0x7feb9c06bbd0 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feba1a5a700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feb9c109510 0x7feb9c06e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feba1259700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9c06bbd0 0x7feb9c074cd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7feb8c00b530 tx=0x7feb8c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feb927fc700 1 -- 172.21.15.67:0/1280337105 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c015020 con 0x7feb9c06bbd0 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feba34bd700 1 -- 172.21.15.67:0/1280337105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb9c072800 con 0x7feb9c06bbd0 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feb927fc700 1 -- 172.21.15.67:0/1280337105 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb8c011ce0 con 0x7feb9c06bbd0 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feb927fc700 1 -- 172.21.15.67:0/1280337105 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c010e40 con 0x7feb9c06bbd0 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.344+0000 7feba34bd700 1 -- 172.21.15.67:0/1280337105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb9c072d40 con 0x7feb9c06bbd0 2024-05-03T17:22:13.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.345+0000 7feba34bd700 1 -- 172.21.15.67:0/1280337105 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feb9c0732d0 con 0x7feb9c06bbd0 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.347+0000 7feb927fc700 1 -- 172.21.15.67:0/1280337105 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7feb8c018020 con 0x7feb9c06bbd0 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.347+0000 7feb927fc700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7feb880b2910 0x7feb880b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.348+0000 7feba0a58700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7feb880b2910 0x7feb880b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.348+0000 7feb927fc700 1 -- 172.21.15.67:0/1280337105 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7feb8c0e2250 con 0x7feb9c06bbd0 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.348+0000 7feb927fc700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264] conn(0x7feb880bc4d0 0x7feb880be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.348+0000 7feb927fc700 1 -- 172.21.15.67:0/1280337105 --> [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb8c01be50 con 0x7feb880bc4d0 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.348+0000 7feb927fc700 1 -- 172.21.15.67:0/1280337105 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7feb8c0e26a0 con 0x7feb9c06bbd0 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.348+0000 7feba0a58700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7feb880b2910 0x7feb880b4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7feb9c06f520 tx=0x7feb98009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.348+0000 7feba1a5a700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264] conn(0x7feb880bc4d0 0x7feb880be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.349+0000 7feba1a5a700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264] conn(0x7feb880bc4d0 0x7feb880be970 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.349+0000 7feb927fc700 1 -- 172.21.15.67:0/1280337105 <== osd.2 v2:172.21.15.82:6800/1888803264 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7feb8c01be50 con 0x7feb880bc4d0 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.367+0000 7feba34bd700 1 -- 172.21.15.67:0/1280337105 --> [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7feb9c104bd0 con 0x7feb880bc4d0 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.368+0000 7feb927fc700 1 -- 172.21.15.67:0/1280337105 <== osd.2 v2:172.21.15.82:6800/1888803264 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7feb9c104bd0 con 0x7feb880bc4d0 2024-05-03T17:22:13.375 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.368+0000 7feba34bd700 1 -- 172.21.15.67:0/1280337105 >> [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264] conn(0x7feb880bc4d0 msgr2=0x7feb880be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.368+0000 7feba34bd700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264] conn(0x7feb880bc4d0 0x7feb880be970 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.368+0000 7feba34bd700 1 -- 172.21.15.67:0/1280337105 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7feb880b2910 msgr2=0x7feb880b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.368+0000 7feba34bd700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7feb880b2910 0x7feb880b4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7feb9c06f520 tx=0x7feb98009190 comp rx=0 tx=0).stop 2024-05-03T17:22:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.368+0000 7feba34bd700 1 -- 172.21.15.67:0/1280337105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9c06bbd0 msgr2=0x7feb9c074cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.368+0000 7feba34bd700 1 --2- 172.21.15.67:0/1280337105 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feb9c06bbd0 0x7feb9c074cd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7feb8c00b530 tx=0x7feb8c00b840 comp rx=0 tx=0).stop 2024-05-03T17:22:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.368+0000 7feba1a5a700 1 -- 172.21.15.67:0/1280337105 reap_dead start 2024-05-03T17:22:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.368+0000 7feba34bd700 1 -- 172.21.15.67:0/1280337105 shutdown_connections 2024-05-03T17:22:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.368+0000 7feba34bd700 1 -- 172.21.15.67:0/1280337105 >> 172.21.15.67:0/1280337105 conn(0x7feb9c079350 msgr2=0x7feb9c101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.369+0000 7feba34bd700 1 -- 172.21.15.67:0/1280337105 shutdown_connections 2024-05-03T17:22:13.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:12.369+0000 7feba34bd700 1 -- 172.21.15.67:0/1280337105 wait complete. 2024-05-03T17:22:14.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:14 smithi070 ceph-mon[33250]: pgmap v344: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:22:14.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:14 smithi067 ceph-mon[29731]: pgmap v344: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:22:14.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:14 smithi082 ceph-mon[33535]: pgmap v344: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:22:14.654 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:16.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:16 smithi070 ceph-mon[33250]: pgmap v345: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:22:16.392 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:16 smithi067 ceph-mon[29731]: pgmap v345: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:22:16.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:16 smithi082 ceph-mon[33535]: pgmap v345: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:22:18.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:18 smithi070 ceph-mon[33250]: pgmap v346: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:18.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:18 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1774080878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-03T17:22:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.200+0000 7f65dc334700 1 -- 172.21.15.67:0/1445478432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65d4102620 msgr2=0x7f65d4102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:18.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.200+0000 7f65dc334700 1 --2- 172.21.15.67:0/1445478432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65d4102620 0x7f65d4102a00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f65d00099c0 tx=0x7f65d0009cd0 comp rx=0 tx=0).stop 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.200+0000 7f65dc334700 1 -- 172.21.15.67:0/1445478432 shutdown_connections 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.200+0000 7f65dc334700 1 --2- 172.21.15.67:0/1445478432 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d4107700 0x7f65d410f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.200+0000 7f65dc334700 1 --2- 172.21.15.67:0/1445478432 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65d4102620 0x7f65d4102a00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.200+0000 7f65dc334700 1 --2- 172.21.15.67:0/1445478432 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d4102fd0 0x7f65d4107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.200+0000 7f65dc334700 1 -- 172.21.15.67:0/1445478432 >> 172.21.15.67:0/1445478432 conn(0x7f65d4079440 msgr2=0x7f65d4079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.200+0000 7f65dc334700 1 -- 172.21.15.67:0/1445478432 shutdown_connections 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.200+0000 7f65dc334700 1 -- 172.21.15.67:0/1445478432 wait complete. 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65dc334700 1 Processor -- start 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65dc334700 1 -- start start 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65dc334700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d4102620 0x7f65d4198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65dc334700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d4102fd0 0x7f65d4198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65dc334700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65d4107700 0x7f65d41992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:18.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65dc334700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f65d4112610 con 0x7f65d4107700 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65dc334700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f65d4112490 con 0x7f65d4102620 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65dc334700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f65d4112790 con 0x7f65d4102fd0 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65da8d1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65d4107700 0x7f65d41992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65da8d1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65d4107700 0x7f65d41992c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45760/0 (socket says 172.21.15.67:45760) 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65da8d1700 1 -- 172.21.15.67:0/1774080878 learned_addr learned my addr 172.21.15.67:0/1774080878 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65d98cf700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d4102fd0 0x7f65d4198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65da0d0700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d4102620 0x7f65d4198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65da8d1700 1 -- 172.21.15.67:0/1774080878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d4102fd0 msgr2=0x7f65d4198d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65da8d1700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d4102fd0 0x7f65d4198d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65da8d1700 1 -- 172.21.15.67:0/1774080878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d4102620 msgr2=0x7f65d4198840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65da8d1700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d4102620 0x7f65d4198840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65da8d1700 1 -- 172.21.15.67:0/1774080878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65d41a80b0 con 0x7f65d4107700 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.201+0000 7f65da0d0700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d4102620 0x7f65d4198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.202+0000 7f65da8d1700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65d4107700 0x7f65d41992c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00b970 tx=0x7f65cc00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.202+0000 7f65cb7fe700 1 -- 172.21.15.67:0/1774080878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc019070 con 0x7f65d4107700 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.202+0000 7f65cb7fe700 1 -- 172.21.15.67:0/1774080878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65cc00cde0 con 0x7f65d4107700 2024-05-03T17:22:18.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.202+0000 7f65dc334700 1 -- 172.21.15.67:0/1774080878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65d41a83a0 con 0x7f65d4107700 2024-05-03T17:22:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.202+0000 7f65cb7fe700 1 -- 172.21.15.67:0/1774080878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65cc007370 con 0x7f65d4107700 2024-05-03T17:22:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.202+0000 7f65dc334700 1 -- 172.21.15.67:0/1774080878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65d41a8930 con 0x7f65d4107700 2024-05-03T17:22:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.203+0000 7f65dc334700 1 -- 172.21.15.67:0/1774080878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65d406a190 con 0x7f65d4107700 2024-05-03T17:22:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.204+0000 7f65cb7fe700 1 -- 172.21.15.67:0/1774080878 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f65cc01c070 con 0x7f65d4107700 2024-05-03T17:22:18.424 INFO:teuthology.orchestra.run.smithi067.stdout:257698037807 2024-05-03T17:22:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.205+0000 7f65cb7fe700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f65c00b2990 0x7f65c00b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.206+0000 7f65da0d0700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f65c00b2990 0x7f65c00b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:18.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.206+0000 7f65cb7fe700 1 -- 172.21.15.67:0/1774080878 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f65cc0e1e10 con 0x7f65d4107700 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.206+0000 7f65da0d0700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f65c00b2990 0x7f65c00b4e50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009fd0 tx=0x7f65d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.209+0000 7f65cb7fe700 1 -- 172.21.15.67:0/1774080878 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f65cc023070 con 0x7f65d4107700 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.431+0000 7f65dc334700 1 -- 172.21.15.67:0/1774080878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f65d402e510 con 0x7f65d4107700 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.431+0000 7f65cb7fe700 1 -- 172.21.15.67:0/1774080878 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f65cc0a9b50 con 0x7f65d4107700 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.435+0000 7f65dc334700 1 -- 172.21.15.67:0/1774080878 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f65c00b2990 msgr2=0x7f65c00b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.435+0000 7f65dc334700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f65c00b2990 0x7f65c00b4e50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f65d0009fd0 tx=0x7f65d0005fd0 comp rx=0 tx=0).stop 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.435+0000 7f65dc334700 1 -- 172.21.15.67:0/1774080878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65d4107700 msgr2=0x7f65d41992c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.435+0000 7f65dc334700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65d4107700 0x7f65d41992c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f65cc00b970 tx=0x7f65cc00bc80 comp rx=0 tx=0).stop 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.435+0000 7f65dc334700 1 -- 172.21.15.67:0/1774080878 shutdown_connections 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.435+0000 7f65dc334700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f65c00b2990 0x7f65c00b4e50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.435+0000 7f65dc334700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f65d4107700 0x7f65d41992c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.435+0000 7f65dc334700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f65d4102620 0x7f65d4198840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.435+0000 7f65dc334700 1 --2- 172.21.15.67:0/1774080878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f65d4102fd0 0x7f65d4198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.435+0000 7f65dc334700 1 -- 172.21.15.67:0/1774080878 >> 172.21.15.67:0/1774080878 conn(0x7f65d4079440 msgr2=0x7f65d4100350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:18.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.436+0000 7f65dc334700 1 -- 172.21.15.67:0/1774080878 shutdown_connections 2024-05-03T17:22:18.426 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:17.436+0000 7f65dc334700 1 -- 172.21.15.67:0/1774080878 wait complete. 2024-05-03T17:22:18.426 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:18 smithi067 ceph-mon[29731]: pgmap v346: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:18.426 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:18 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1774080878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-03T17:22:18.517 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:18 smithi082 ceph-mon[33535]: pgmap v346: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:18.518 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:18 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1774080878' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-03T17:22:18.759 INFO:teuthology.orchestra.run.smithi067.stdout:158913790019 2024-05-03T17:22:18.759 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-03T17:22:18.864 INFO:teuthology.orchestra.run.smithi067.stdout:77309411411 2024-05-03T17:22:18.865 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-03T17:22:20.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:20 smithi070 ceph-mon[33250]: pgmap v347: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:22:20.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:20 smithi067 ceph-mon[29731]: pgmap v347: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:22:20.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:20 smithi082 ceph-mon[33535]: pgmap v347: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:22:22.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:22 smithi070 ceph-mon[33250]: pgmap v348: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:22:22.316 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:22 smithi082 ceph-mon[33535]: pgmap v348: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:22:22.317 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:22 smithi067 ceph-mon[29731]: pgmap v348: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:22:22.324 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:22.922 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037803 got 257698037807 for osd.9 2024-05-03T17:22:22.922 DEBUG:teuthology.parallel:result is None 2024-05-03T17:22:23.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.136+0000 7f0ac943a700 1 -- 172.21.15.67:0/3722014775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac410a9e0 msgr2=0x7f0ac410ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:23.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.136+0000 7f0ac943a700 1 --2- 172.21.15.67:0/3722014775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac410a9e0 0x7f0ac410ae60 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0ab4009990 tx=0x7f0ab4009ca0 comp rx=0 tx=0).stop 2024-05-03T17:22:23.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.136+0000 7f0ac943a700 1 -- 172.21.15.67:0/3722014775 shutdown_connections 2024-05-03T17:22:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.136+0000 7f0ac943a700 1 --2- 172.21.15.67:0/3722014775 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac411cc80 0x7f0ac411f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.136+0000 7f0ac943a700 1 --2- 172.21.15.67:0/3722014775 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ac4111c80 0x7f0ac410a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.136+0000 7f0ac943a700 1 --2- 172.21.15.67:0/3722014775 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac410a9e0 0x7f0ac410ae60 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.136+0000 7f0ac943a700 1 -- 172.21.15.67:0/3722014775 >> 172.21.15.67:0/3722014775 conn(0x7f0ac406e610 msgr2=0x7f0ac406ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.136+0000 7f0ac943a700 1 -- 172.21.15.67:0/3722014775 shutdown_connections 2024-05-03T17:22:23.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.136+0000 7f0ac943a700 1 -- 172.21.15.67:0/3722014775 wait complete. 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac943a700 1 Processor -- start 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac943a700 1 -- start start 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac943a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac410a9e0 0x7f0ac41ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac943a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ac4111c80 0x7f0ac41aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac943a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac411cc80 0x7f0ac41a87f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac943a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac4122510 con 0x7f0ac410a9e0 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac943a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac4122390 con 0x7f0ac411cc80 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac943a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0ac4122690 con 0x7f0ac4111c80 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac410a9e0 0x7f0ac41ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac410a9e0 0x7f0ac41ae670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45780/0 (socket says 172.21.15.67:45780) 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac2ffd700 1 -- 172.21.15.67:0/2775661371 learned_addr learned my addr 172.21.15.67:0/2775661371 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac37fe700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac411cc80 0x7f0ac41a87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:23.140 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.137+0000 7f0ac27fc700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ac4111c80 0x7f0ac41aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.138+0000 7f0ac2ffd700 1 -- 172.21.15.67:0/2775661371 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ac4111c80 msgr2=0x7f0ac41aebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.138+0000 7f0ac2ffd700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ac4111c80 0x7f0ac41aebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.138+0000 7f0ac2ffd700 1 -- 172.21.15.67:0/2775661371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac411cc80 msgr2=0x7f0ac41a87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.138+0000 7f0ac2ffd700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac411cc80 0x7f0ac41a87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.138+0000 7f0ac2ffd700 1 -- 172.21.15.67:0/2775661371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ac41a8fd0 con 0x7f0ac410a9e0 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.138+0000 7f0ac27fc700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ac4111c80 0x7f0ac41aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.138+0000 7f0ac2ffd700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac410a9e0 0x7f0ac41ae670 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800e9c0 tx=0x7f0ab800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.138+0000 7f0aabfff700 1 -- 172.21.15.67:0/2775661371 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab800cbd0 con 0x7f0ac410a9e0 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.138+0000 7f0ac943a700 1 -- 172.21.15.67:0/2775661371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ac41a92c0 con 0x7f0ac410a9e0 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.138+0000 7f0ac943a700 1 -- 172.21.15.67:0/2775661371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ac41b54d0 con 0x7f0ac410a9e0 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.138+0000 7f0aabfff700 1 -- 172.21.15.67:0/2775661371 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ab800cd70 con 0x7f0ac410a9e0 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.138+0000 7f0aabfff700 1 -- 172.21.15.67:0/2775661371 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8005610 con 0x7f0ac410a9e0 2024-05-03T17:22:23.141 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.139+0000 7f0ac943a700 1 -- 172.21.15.67:0/2775661371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ac406fba0 con 0x7f0ac410a9e0 2024-05-03T17:22:23.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.141+0000 7f0aabfff700 1 -- 172.21.15.67:0/2775661371 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0ab8004750 con 0x7f0ac410a9e0 2024-05-03T17:22:23.142 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.141+0000 7f0aabfff700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0aac0b2990 0x7f0aac0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:23.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.141+0000 7f0ac27fc700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0aac0b2990 0x7f0aac0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:23.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.141+0000 7f0aabfff700 1 -- 172.21.15.67:0/2775661371 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f0ab80e28e0 con 0x7f0ac410a9e0 2024-05-03T17:22:23.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.142+0000 7f0ac27fc700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0aac0b2990 0x7f0aac0b4e50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400b2e0 tx=0x7f0ab401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:23.146 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.145+0000 7f0aabfff700 1 -- 172.21.15.67:0/2775661371 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f0ab8054d70 con 0x7f0ac410a9e0 2024-05-03T17:22:23.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.365+0000 7f0ac943a700 1 -- 172.21.15.67:0/2775661371 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f0ac402e510 con 0x7f0ac410a9e0 2024-05-03T17:22:23.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.365+0000 7f0aabfff700 1 -- 172.21.15.67:0/2775661371 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f0ab80aa490 con 0x7f0ac410a9e0 2024-05-03T17:22:23.367 INFO:teuthology.orchestra.run.smithi067.stdout:111669149774 2024-05-03T17:22:23.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.369+0000 7f0ac943a700 1 -- 172.21.15.67:0/2775661371 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0aac0b2990 msgr2=0x7f0aac0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:23.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.369+0000 7f0ac943a700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0aac0b2990 0x7f0aac0b4e50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400b2e0 tx=0x7f0ab401a040 comp rx=0 tx=0).stop 2024-05-03T17:22:23.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.369+0000 7f0ac943a700 1 -- 172.21.15.67:0/2775661371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac410a9e0 msgr2=0x7f0ac41ae670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:23.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.369+0000 7f0ac943a700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac410a9e0 0x7f0ac41ae670 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800e9c0 tx=0x7f0ab800ecd0 comp rx=0 tx=0).stop 2024-05-03T17:22:23.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.369+0000 7f0ac943a700 1 -- 172.21.15.67:0/2775661371 shutdown_connections 2024-05-03T17:22:23.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.369+0000 7f0ac943a700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f0aac0b2990 0x7f0aac0b4e50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:23.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.369+0000 7f0ac943a700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f0ac411cc80 0x7f0ac41a87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:23.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.369+0000 7f0ac943a700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0ac410a9e0 0x7f0ac41ae670 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:23.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.369+0000 7f0ac943a700 1 --2- 172.21.15.67:0/2775661371 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0ac4111c80 0x7f0ac41aebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:23.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.369+0000 7f0ac943a700 1 -- 172.21.15.67:0/2775661371 >> 172.21.15.67:0/2775661371 conn(0x7f0ac406e610 msgr2=0x7f0ac411af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:23.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.370+0000 7f0ac943a700 1 -- 172.21.15.67:0/2775661371 shutdown_connections 2024-05-03T17:22:23.372 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.370+0000 7f0ac943a700 1 -- 172.21.15.67:0/2775661371 wait complete. 2024-05-03T17:22:24.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.132+0000 7ffa1569a700 1 -- 172.21.15.67:0/1294391628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa10101a90 msgr2=0x7ffa10101f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:24.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.132+0000 7ffa1569a700 1 --2- 172.21.15.67:0/1294391628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa10101a90 0x7ffa10101f10 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009990 tx=0x7ffa04009ca0 comp rx=0 tx=0).stop 2024-05-03T17:22:24.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.132+0000 7ffa1569a700 1 -- 172.21.15.67:0/1294391628 shutdown_connections 2024-05-03T17:22:24.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.132+0000 7ffa1569a700 1 --2- 172.21.15.67:0/1294391628 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa10102450 0x7ffa10110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:24.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.132+0000 7ffa1569a700 1 --2- 172.21.15.67:0/1294391628 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa10109520 0x7ffa10109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:24.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.132+0000 7ffa1569a700 1 --2- 172.21.15.67:0/1294391628 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa10101a90 0x7ffa10101f10 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:24.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.132+0000 7ffa1569a700 1 -- 172.21.15.67:0/1294391628 >> 172.21.15.67:0/1294391628 conn(0x7ffa10078a10 msgr2=0x7ffa100ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:24.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.132+0000 7ffa1569a700 1 -- 172.21.15.67:0/1294391628 shutdown_connections 2024-05-03T17:22:24.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.132+0000 7ffa1569a700 1 -- 172.21.15.67:0/1294391628 wait complete. 2024-05-03T17:22:24.134 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.132+0000 7ffa1569a700 1 Processor -- start 2024-05-03T17:22:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa1569a700 1 -- start start 2024-05-03T17:22:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa1569a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa10101a90 0x7ffa10102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa1569a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa10102450 0x7ffa101031b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa1569a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa10109520 0x7ffa101036f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa1569a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ffa101136a0 con 0x7ffa10101a90 2024-05-03T17:22:24.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa1569a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ffa10113520 con 0x7ffa10102450 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa1569a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ffa10113820 con 0x7ffa10109520 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa0effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa10101a90 0x7ffa10102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa0effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa10101a90 0x7ffa10102c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45808/0 (socket says 172.21.15.67:45808) 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa0effd700 1 -- 172.21.15.67:0/3982889883 learned_addr learned my addr 172.21.15.67:0/3982889883 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa0e7fc700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa10102450 0x7ffa101031b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa0f7fe700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa10109520 0x7ffa101036f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa0effd700 1 -- 172.21.15.67:0/3982889883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa10109520 msgr2=0x7ffa101036f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa0effd700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa10109520 0x7ffa101036f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa0effd700 1 -- 172.21.15.67:0/3982889883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa10102450 msgr2=0x7ffa101031b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa0effd700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa10102450 0x7ffa101031b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.133+0000 7ffa0effd700 1 -- 172.21.15.67:0/3982889883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa101a80e0 con 0x7ffa10101a90 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.134+0000 7ffa0effd700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa10101a90 0x7ffa10102c70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000d950 tx=0x7ffa0000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.134+0000 7ff9f7fff700 1 -- 172.21.15.67:0/3982889883 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa00016070 con 0x7ffa10101a90 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.134+0000 7ffa1569a700 1 -- 172.21.15.67:0/3982889883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa101a83d0 con 0x7ffa10101a90 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.134+0000 7ff9f7fff700 1 -- 172.21.15.67:0/3982889883 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa00009df0 con 0x7ffa10101a90 2024-05-03T17:22:24.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.134+0000 7ffa1569a700 1 -- 172.21.15.67:0/3982889883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa101a8960 con 0x7ffa10101a90 2024-05-03T17:22:24.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.134+0000 7ff9f7fff700 1 -- 172.21.15.67:0/3982889883 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0000f3c0 con 0x7ffa10101a90 2024-05-03T17:22:24.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.135+0000 7ffa1569a700 1 -- 172.21.15.67:0/3982889883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9fc005330 con 0x7ffa10101a90 2024-05-03T17:22:24.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.136+0000 7ff9f7fff700 1 -- 172.21.15.67:0/3982889883 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ffa00019070 con 0x7ffa10101a90 2024-05-03T17:22:24.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.137+0000 7ff9f7fff700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff9f80b2990 0x7ff9f80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:24.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.137+0000 7ffa0e7fc700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff9f80b2990 0x7ff9f80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:24.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.137+0000 7ff9f7fff700 1 -- 172.21.15.67:0/3982889883 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7ffa000e1d70 con 0x7ffa10101a90 2024-05-03T17:22:24.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.138+0000 7ffa0e7fc700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff9f80b2990 0x7ff9f80b4e50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffa10108c10 tx=0x7ffa04005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:24.143 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.142+0000 7ff9f7fff700 1 -- 172.21.15.67:0/3982889883 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ffa000a9920 con 0x7ffa10101a90 2024-05-03T17:22:24.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:24 smithi067 ceph-mon[29731]: pgmap v349: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-05-03T17:22:24.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:24 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2775661371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-03T17:22:24.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:24 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/563273991' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-03T17:22:24.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:24 smithi070 ceph-mon[33250]: pgmap v349: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-05-03T17:22:24.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:24 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2775661371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-03T17:22:24.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:24 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/563273991' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-03T17:22:24.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.373+0000 7ffa1569a700 1 -- 172.21.15.67:0/3982889883 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7ff9fc0058f0 con 0x7ffa10101a90 2024-05-03T17:22:24.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.374+0000 7ff9f7fff700 1 -- 172.21.15.67:0/3982889883 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7ffa000a9220 con 0x7ffa10101a90 2024-05-03T17:22:24.375 INFO:teuthology.orchestra.run.smithi067.stdout:304942678054 2024-05-03T17:22:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.376+0000 7ffa1569a700 1 -- 172.21.15.67:0/3982889883 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff9f80b2990 msgr2=0x7ff9f80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.376+0000 7ffa1569a700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff9f80b2990 0x7ff9f80b4e50 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffa10108c10 tx=0x7ffa04005fd0 comp rx=0 tx=0).stop 2024-05-03T17:22:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.376+0000 7ffa1569a700 1 -- 172.21.15.67:0/3982889883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa10101a90 msgr2=0x7ffa10102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.376+0000 7ffa1569a700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa10101a90 0x7ffa10102c70 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ffa0000d950 tx=0x7ffa0000dc60 comp rx=0 tx=0).stop 2024-05-03T17:22:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.376+0000 7ffa1569a700 1 -- 172.21.15.67:0/3982889883 shutdown_connections 2024-05-03T17:22:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.376+0000 7ffa1569a700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff9f80b2990 0x7ff9f80b4e50 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.376+0000 7ffa1569a700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ffa10109520 0x7ffa101036f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.376+0000 7ffa1569a700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ffa10101a90 0x7ffa10102c70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.376+0000 7ffa1569a700 1 --2- 172.21.15.67:0/3982889883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ffa10102450 0x7ffa101031b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:24.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.376+0000 7ffa1569a700 1 -- 172.21.15.67:0/3982889883 >> 172.21.15.67:0/3982889883 conn(0x7ffa10078a10 msgr2=0x7ffa10100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:24.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.376+0000 7ffa1569a700 1 -- 172.21.15.67:0/3982889883 shutdown_connections 2024-05-03T17:22:24.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:24.376+0000 7ffa1569a700 1 -- 172.21.15.67:0/3982889883 wait complete. 2024-05-03T17:22:24.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:24 smithi082 ceph-mon[33535]: pgmap v349: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-05-03T17:22:24.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:24 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2775661371' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-03T17:22:24.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:24 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/563273991' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-03T17:22:24.833 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149769 got 111669149774 for osd.3 2024-05-03T17:22:24.833 DEBUG:teuthology.parallel:result is None 2024-05-03T17:22:25.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:25 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3982889883' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-03T17:22:25.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:25 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3982889883' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-03T17:22:25.499 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678049 got 304942678054 for osd.11 2024-05-03T17:22:25.500 DEBUG:teuthology.parallel:result is None 2024-05-03T17:22:25.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:25 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3982889883' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-03T17:22:26.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:26 smithi070 ceph-mon[33250]: pgmap v350: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-05-03T17:22:26.386 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:26 smithi067 ceph-mon[29731]: pgmap v350: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-05-03T17:22:26.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.567+0000 7f816036c700 1 -- 172.21.15.67:0/3705675088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8158102d50 msgr2=0x7f815810f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:26.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.567+0000 7f816036c700 1 --2- 172.21.15.67:0/3705675088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8158102d50 0x7f815810f240 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f815000b0a0 tx=0x7f815000b3b0 comp rx=0 tx=0).stop 2024-05-03T17:22:26.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.567+0000 7f816036c700 1 -- 172.21.15.67:0/3705675088 shutdown_connections 2024-05-03T17:22:26.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.567+0000 7f816036c700 1 --2- 172.21.15.67:0/3705675088 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8158102d50 0x7f815810f240 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:26.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.567+0000 7f816036c700 1 --2- 172.21.15.67:0/3705675088 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8158108400 0x7f81581087e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:26.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.567+0000 7f816036c700 1 --2- 172.21.15.67:0/3705675088 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8158102390 0x7f8158102810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.567+0000 7f816036c700 1 -- 172.21.15.67:0/3705675088 >> 172.21.15.67:0/3705675088 conn(0x7f8158079310 msgr2=0x7f8158079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.567+0000 7f816036c700 1 -- 172.21.15.67:0/3705675088 shutdown_connections 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.567+0000 7f816036c700 1 -- 172.21.15.67:0/3705675088 wait complete. 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.567+0000 7f816036c700 1 Processor -- start 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.567+0000 7f816036c700 1 -- start start 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f816036c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8158102390 0x7f815819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f816036c700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8158102d50 0x7f815819ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f816036c700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8158108400 0x7f81581a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f816036c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8158112550 con 0x7f8158102390 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f816036c700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f81581123d0 con 0x7f8158102d50 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f816036c700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f81581126d0 con 0x7f8158108400 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f815e108700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8158102390 0x7f815819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f815e108700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8158102390 0x7f815819a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45796/0 (socket says 172.21.15.67:45796) 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f815e108700 1 -- 172.21.15.67:0/563273991 learned_addr learned my addr 172.21.15.67:0/563273991 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f815d907700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8158102d50 0x7f815819ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:26.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f815e909700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8158108400 0x7f81581a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:26.505 INFO:teuthology.orchestra.run.smithi067.stdout:231928234038 2024-05-03T17:22:26.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f815e108700 1 -- 172.21.15.67:0/563273991 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8158108400 msgr2=0x7f81581a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:26.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f815e108700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8158108400 0x7f81581a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:26.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f815e108700 1 -- 172.21.15.67:0/563273991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8158102d50 msgr2=0x7f815819ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:26.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f815e108700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8158102d50 0x7f815819ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:26.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f815e108700 1 -- 172.21.15.67:0/563273991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81581a5f10 con 0x7f8158102390 2024-05-03T17:22:26.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f815e909700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8158108400 0x7f81581a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:22:26.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f815e108700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8158102390 0x7f815819a950 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f815400eb30 tx=0x7f815400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f814f7fe700 1 -- 172.21.15.67:0/563273991 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815400cc90 con 0x7f8158102390 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f816036c700 1 -- 172.21.15.67:0/563273991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81581a6140 con 0x7f8158102390 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f814f7fe700 1 -- 172.21.15.67:0/563273991 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81540044d0 con 0x7f8158102390 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f816036c700 1 -- 172.21.15.67:0/563273991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81581a66a0 con 0x7f8158102390 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.568+0000 7f814f7fe700 1 -- 172.21.15.67:0/563273991 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8154010430 con 0x7f8158102390 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.569+0000 7f816036c700 1 -- 172.21.15.67:0/563273991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f815806a190 con 0x7f8158102390 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.572+0000 7f814f7fe700 1 -- 172.21.15.67:0/563273991 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f8154003ea0 con 0x7f8158102390 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.572+0000 7f814f7fe700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f81440b2910 0x7f81440b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.573+0000 7f815d907700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f81440b2910 0x7f81440b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.573+0000 7f814f7fe700 1 -- 172.21.15.67:0/563273991 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f81540e2780 con 0x7f8158102390 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.573+0000 7f815d907700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f81440b2910 0x7f81440b4dd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8158079980 tx=0x7f8148009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.576+0000 7f814f7fe700 1 -- 172.21.15.67:0/563273991 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f8154054d70 con 0x7f8158102390 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.795+0000 7f816036c700 1 -- 172.21.15.67:0/563273991 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f815806c000 con 0x7f8158102390 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.795+0000 7f814f7fe700 1 -- 172.21.15.67:0/563273991 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f81540aa330 con 0x7f8158102390 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.798+0000 7f816036c700 1 -- 172.21.15.67:0/563273991 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f81440b2910 msgr2=0x7f81440b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:26.506 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.798+0000 7f816036c700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f81440b2910 0x7f81440b4dd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f8158079980 tx=0x7f8148009260 comp rx=0 tx=0).stop 2024-05-03T17:22:26.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.798+0000 7f816036c700 1 -- 172.21.15.67:0/563273991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8158102390 msgr2=0x7f815819a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:26.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.798+0000 7f816036c700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8158102390 0x7f815819a950 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f815400eb30 tx=0x7f815400ee40 comp rx=0 tx=0).stop 2024-05-03T17:22:26.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.798+0000 7f816036c700 1 -- 172.21.15.67:0/563273991 shutdown_connections 2024-05-03T17:22:26.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.798+0000 7f816036c700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f81440b2910 0x7f81440b4dd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:26.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.798+0000 7f816036c700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8158108400 0x7f81581a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:26.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.798+0000 7f816036c700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8158102390 0x7f815819a950 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:26.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.798+0000 7f816036c700 1 --2- 172.21.15.67:0/563273991 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8158102d50 0x7f815819ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:26.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.798+0000 7f816036c700 1 -- 172.21.15.67:0/563273991 >> 172.21.15.67:0/563273991 conn(0x7f8158079310 msgr2=0x7f8158100ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:26.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.798+0000 7f816036c700 1 -- 172.21.15.67:0/563273991 shutdown_connections 2024-05-03T17:22:26.507 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:23.798+0000 7f816036c700 1 -- 172.21.15.67:0/563273991 wait complete. 2024-05-03T17:22:26.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:26 smithi082 ceph-mon[33535]: pgmap v350: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-05-03T17:22:27.177 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234033 got 231928234038 for osd.8 2024-05-03T17:22:27.177 DEBUG:teuthology.parallel:result is None 2024-05-03T17:22:27.373 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.373+0000 7f49d0e91700 1 -- 172.21.15.67:0/2593309134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49cc075ff0 msgr2=0x7f49cc10f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.373+0000 7f49d0e91700 1 --2- 172.21.15.67:0/2593309134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49cc075ff0 0x7f49cc10f9d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009990 tx=0x7f49c0009ca0 comp rx=0 tx=0).stop 2024-05-03T17:22:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.373+0000 7f49d0e91700 1 -- 172.21.15.67:0/2593309134 shutdown_connections 2024-05-03T17:22:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.373+0000 7f49d0e91700 1 --2- 172.21.15.67:0/2593309134 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49cc10ff10 0x7f49cc112300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.373+0000 7f49d0e91700 1 --2- 172.21.15.67:0/2593309134 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49cc0756d0 0x7f49cc075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.373+0000 7f49d0e91700 1 --2- 172.21.15.67:0/2593309134 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49cc075ff0 0x7f49cc10f9d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.373+0000 7f49d0e91700 1 -- 172.21.15.67:0/2593309134 >> 172.21.15.67:0/2593309134 conn(0x7f49cc07a660 msgr2=0x7f49cc07aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.373+0000 7f49d0e91700 1 -- 172.21.15.67:0/2593309134 shutdown_connections 2024-05-03T17:22:27.374 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.373+0000 7f49d0e91700 1 -- 172.21.15.67:0/2593309134 wait complete. 2024-05-03T17:22:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49d0e91700 1 Processor -- start 2024-05-03T17:22:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49d0e91700 1 -- start start 2024-05-03T17:22:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49d0e91700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49cc0756d0 0x7f49cc19bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49d0e91700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49cc075ff0 0x7f49cc19c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49d0e91700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49cc10ff10 0x7f49cc1a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49d0e91700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f49cc115850 con 0x7f49cc0756d0 2024-05-03T17:22:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49d0e91700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f49cc1156d0 con 0x7f49cc075ff0 2024-05-03T17:22:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49d0e91700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f49cc1159d0 con 0x7f49cc10ff10 2024-05-03T17:22:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49ca59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49cc0756d0 0x7f49cc19bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49ca59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49cc0756d0 0x7f49cc19bea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50732/0 (socket says 172.21.15.67:50732) 2024-05-03T17:22:27.376 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49ca59c700 1 -- 172.21.15.67:0/2558381829 learned_addr learned my addr 172.21.15.67:0/2558381829 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49c9d9b700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49cc075ff0 0x7f49cc19c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49cad9d700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49cc10ff10 0x7f49cc1a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49ca59c700 1 -- 172.21.15.67:0/2558381829 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49cc10ff10 msgr2=0x7f49cc1a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49ca59c700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49cc10ff10 0x7f49cc1a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49ca59c700 1 -- 172.21.15.67:0/2558381829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49cc075ff0 msgr2=0x7f49cc19c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49ca59c700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49cc075ff0 0x7f49cc19c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.374+0000 7f49ca59c700 1 -- 172.21.15.67:0/2558381829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49cc1a0eb0 con 0x7f49cc0756d0 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.375+0000 7f49ca59c700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49cc0756d0 0x7f49cc19bea0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00d980 tx=0x7f49bc00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.375+0000 7f49bb7fe700 1 -- 172.21.15.67:0/2558381829 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc017070 con 0x7f49cc0756d0 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.375+0000 7f49d0e91700 1 -- 172.21.15.67:0/2558381829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49cc1a11a0 con 0x7f49cc0756d0 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.375+0000 7f49bb7fe700 1 -- 172.21.15.67:0/2558381829 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49bc004830 con 0x7f49cc0756d0 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.375+0000 7f49bb7fe700 1 -- 172.21.15.67:0/2558381829 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49bc005440 con 0x7f49cc0756d0 2024-05-03T17:22:27.377 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.375+0000 7f49d0e91700 1 -- 172.21.15.67:0/2558381829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49cc1a8a20 con 0x7f49cc0756d0 2024-05-03T17:22:27.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.376+0000 7f49d0e91700 1 -- 172.21.15.67:0/2558381829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49cc06a190 con 0x7f49cc0756d0 2024-05-03T17:22:27.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.377+0000 7f49bb7fe700 1 -- 172.21.15.67:0/2558381829 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f49bc0055e0 con 0x7f49cc0756d0 2024-05-03T17:22:27.378 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.378+0000 7f49bb7fe700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f49b40b2910 0x7f49b40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:27.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.378+0000 7f49c9d9b700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f49b40b2910 0x7f49b40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:27.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.378+0000 7f49bb7fe700 1 -- 172.21.15.67:0/2558381829 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f49bc0e2730 con 0x7f49cc0756d0 2024-05-03T17:22:27.379 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.378+0000 7f49c9d9b700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f49b40b2910 0x7f49b40b4dd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f49c0006010 tx=0x7f49c001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:27.382 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.382+0000 7f49bb7fe700 1 -- 172.21.15.67:0/2558381829 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f49bc020020 con 0x7f49cc0756d0 2024-05-03T17:22:27.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.606+0000 7f49d0e91700 1 -- 172.21.15.67:0/2558381829 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f49cc02e510 con 0x7f49cc0756d0 2024-05-03T17:22:27.607 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.606+0000 7f49bb7fe700 1 -- 172.21.15.67:0/2558381829 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f49bc0aa2e0 con 0x7f49cc0756d0 2024-05-03T17:22:27.607 INFO:teuthology.orchestra.run.smithi067.stdout:184683593793 2024-05-03T17:22:27.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.608+0000 7f49d0e91700 1 -- 172.21.15.67:0/2558381829 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f49b40b2910 msgr2=0x7f49b40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:27.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.608+0000 7f49d0e91700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f49b40b2910 0x7f49b40b4dd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f49c0006010 tx=0x7f49c001a040 comp rx=0 tx=0).stop 2024-05-03T17:22:27.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.608+0000 7f49d0e91700 1 -- 172.21.15.67:0/2558381829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49cc0756d0 msgr2=0x7f49cc19bea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:27.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.608+0000 7f49d0e91700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49cc0756d0 0x7f49cc19bea0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f49bc00d980 tx=0x7f49bc00dd40 comp rx=0 tx=0).stop 2024-05-03T17:22:27.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.608+0000 7f49d0e91700 1 -- 172.21.15.67:0/2558381829 shutdown_connections 2024-05-03T17:22:27.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.608+0000 7f49d0e91700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f49b40b2910 0x7f49b40b4dd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:27.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.608+0000 7f49d0e91700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f49cc10ff10 0x7f49cc1a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:27.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.608+0000 7f49d0e91700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f49cc0756d0 0x7f49cc19bea0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:27.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.608+0000 7f49d0e91700 1 --2- 172.21.15.67:0/2558381829 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f49cc075ff0 0x7f49cc19c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:27.609 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.608+0000 7f49d0e91700 1 -- 172.21.15.67:0/2558381829 >> 172.21.15.67:0/2558381829 conn(0x7f49cc07a660 msgr2=0x7f49cc101b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:27.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.609+0000 7f49d0e91700 1 -- 172.21.15.67:0/2558381829 shutdown_connections 2024-05-03T17:22:27.610 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:27.609+0000 7f49d0e91700 1 -- 172.21.15.67:0/2558381829 wait complete. 2024-05-03T17:22:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.060+0000 7f77dc6ec700 1 -- 172.21.15.67:0/232720301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d41023a0 msgr2=0x7f77d4102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.060+0000 7f77dc6ec700 1 --2- 172.21.15.67:0/232720301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d41023a0 0x7f77d4102820 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f77c4009990 tx=0x7f77c4009ca0 comp rx=0 tx=0).stop 2024-05-03T17:22:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.060+0000 7f77dc6ec700 1 -- 172.21.15.67:0/232720301 shutdown_connections 2024-05-03T17:22:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.060+0000 7f77dc6ec700 1 --2- 172.21.15.67:0/232720301 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77d4102d60 0x7f77d410f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.060+0000 7f77dc6ec700 1 --2- 172.21.15.67:0/232720301 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77d4108410 0x7f77d41087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:28.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.060+0000 7f77dc6ec700 1 --2- 172.21.15.67:0/232720301 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d41023a0 0x7f77d4102820 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.060+0000 7f77dc6ec700 1 -- 172.21.15.67:0/232720301 >> 172.21.15.67:0/232720301 conn(0x7f77d4079350 msgr2=0x7f77d4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.060+0000 7f77dc6ec700 1 -- 172.21.15.67:0/232720301 shutdown_connections 2024-05-03T17:22:28.062 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.060+0000 7f77dc6ec700 1 -- 172.21.15.67:0/232720301 wait complete. 2024-05-03T17:22:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.061+0000 7f77dc6ec700 1 Processor -- start 2024-05-03T17:22:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.061+0000 7f77dc6ec700 1 -- start start 2024-05-03T17:22:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.061+0000 7f77dc6ec700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d41023a0 0x7f77d4104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.061+0000 7f77dc6ec700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77d4102d60 0x7f77d41045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.061+0000 7f77dc6ec700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77d4108410 0x7f77d4104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.061+0000 7f77dc6ec700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f77d4112580 con 0x7f77d41023a0 2024-05-03T17:22:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.061+0000 7f77dc6ec700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f77d4112400 con 0x7f77d4102d60 2024-05-03T17:22:28.064 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.061+0000 7f77dc6ec700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f77d4112700 con 0x7f77d4108410 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.061+0000 7f77da488700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d41023a0 0x7f77d4104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.061+0000 7f77da488700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d41023a0 0x7f77d4104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50754/0 (socket says 172.21.15.67:50754) 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.061+0000 7f77da488700 1 -- 172.21.15.67:0/419963891 learned_addr learned my addr 172.21.15.67:0/419963891 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77dac89700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77d4108410 0x7f77d4104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77d9c87700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77d4102d60 0x7f77d41045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77da488700 1 -- 172.21.15.67:0/419963891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77d4108410 msgr2=0x7f77d4104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77da488700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77d4108410 0x7f77d4104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77da488700 1 -- 172.21.15.67:0/419963891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77d4102d60 msgr2=0x7f77d41045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77da488700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77d4102d60 0x7f77d41045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77da488700 1 -- 172.21.15.67:0/419963891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77d41a8130 con 0x7f77d41023a0 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77da488700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d41023a0 0x7f77d4104080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f77d000d980 tx=0x7f77d000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77cb7fe700 1 -- 172.21.15.67:0/419963891 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d0016070 con 0x7f77d41023a0 2024-05-03T17:22:28.065 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77cb7fe700 1 -- 172.21.15.67:0/419963891 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f77d00044d0 con 0x7f77d41023a0 2024-05-03T17:22:28.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77cb7fe700 1 -- 172.21.15.67:0/419963891 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d0004d00 con 0x7f77d41023a0 2024-05-03T17:22:28.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77dc6ec700 1 -- 172.21.15.67:0/419963891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77d41a8420 con 0x7f77d41023a0 2024-05-03T17:22:28.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.062+0000 7f77dc6ec700 1 -- 172.21.15.67:0/419963891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77d41a8960 con 0x7f77d41023a0 2024-05-03T17:22:28.066 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.063+0000 7f77dc6ec700 1 -- 172.21.15.67:0/419963891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77d406a190 con 0x7f77d41023a0 2024-05-03T17:22:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.064+0000 7f77cb7fe700 1 -- 172.21.15.67:0/419963891 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f77d0004020 con 0x7f77d41023a0 2024-05-03T17:22:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.065+0000 7f77cb7fe700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f77c00b2910 0x7f77c00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.065+0000 7f77d9c87700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f77c00b2910 0x7f77c00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.066+0000 7f77cb7fe700 1 -- 172.21.15.67:0/419963891 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f77d00e2760 con 0x7f77d41023a0 2024-05-03T17:22:28.067 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.066+0000 7f77d9c87700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f77c00b2910 0x7f77c00b4dd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f77c400b5c0 tx=0x7f77c4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:28.069 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.069+0000 7f77cb7fe700 1 -- 172.21.15.67:0/419963891 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f77d001f020 con 0x7f77d41023a0 2024-05-03T17:22:28.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.289+0000 7f77dc6ec700 1 -- 172.21.15.67:0/419963891 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f77d41a8c60 con 0x7f77d41023a0 2024-05-03T17:22:28.290 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:28 smithi067 ceph-mon[29731]: pgmap v351: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:28.290 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:28 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2558381829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-03T17:22:28.291 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.289+0000 7f77cb7fe700 1 -- 172.21.15.67:0/419963891 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f77d0055e20 con 0x7f77d41023a0 2024-05-03T17:22:28.291 INFO:teuthology.orchestra.run.smithi067.stdout:34359738463 2024-05-03T17:22:28.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.292+0000 7f77dc6ec700 1 -- 172.21.15.67:0/419963891 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f77c00b2910 msgr2=0x7f77c00b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:28.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.292+0000 7f77dc6ec700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f77c00b2910 0x7f77c00b4dd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f77c400b5c0 tx=0x7f77c4005fd0 comp rx=0 tx=0).stop 2024-05-03T17:22:28.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.292+0000 7f77dc6ec700 1 -- 172.21.15.67:0/419963891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d41023a0 msgr2=0x7f77d4104080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:28.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.292+0000 7f77dc6ec700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d41023a0 0x7f77d4104080 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f77d000d980 tx=0x7f77d000dd40 comp rx=0 tx=0).stop 2024-05-03T17:22:28.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.292+0000 7f77dc6ec700 1 -- 172.21.15.67:0/419963891 shutdown_connections 2024-05-03T17:22:28.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.292+0000 7f77dc6ec700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f77c00b2910 0x7f77c00b4dd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:28.292 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.292+0000 7f77dc6ec700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f77d4108410 0x7f77d4104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:28.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.292+0000 7f77dc6ec700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f77d41023a0 0x7f77d4104080 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:28.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.292+0000 7f77dc6ec700 1 --2- 172.21.15.67:0/419963891 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f77d4102d60 0x7f77d41045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:28.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.292+0000 7f77dc6ec700 1 -- 172.21.15.67:0/419963891 >> 172.21.15.67:0/419963891 conn(0x7f77d4079350 msgr2=0x7f77d4101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:28.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.292+0000 7f77dc6ec700 1 -- 172.21.15.67:0/419963891 shutdown_connections 2024-05-03T17:22:28.293 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:28.292+0000 7f77dc6ec700 1 -- 172.21.15.67:0/419963891 wait complete. 2024-05-03T17:22:28.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:28 smithi070 ceph-mon[33250]: pgmap v351: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:28.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:28 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2558381829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-03T17:22:28.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:28 smithi082 ceph-mon[33535]: pgmap v351: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:28.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:28 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2558381829' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-03T17:22:28.523 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593788 got 184683593793 for osd.6 2024-05-03T17:22:28.524 DEBUG:teuthology.parallel:result is None 2024-05-03T17:22:29.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:29 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/419963891' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-03T17:22:29.347 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:29 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/419963891' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-03T17:22:29.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:29 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/419963891' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-03T17:22:30.044 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738455 got 34359738463 for osd.0 2024-05-03T17:22:30.044 DEBUG:teuthology.parallel:result is None 2024-05-03T17:22:30.144 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:30 smithi067 ceph-mon[29731]: pgmap v352: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:22:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.249+0000 7f692a9af700 1 -- 172.21.15.67:0/3809087916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6924102690 msgr2=0x7f6924102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.249+0000 7f692a9af700 1 --2- 172.21.15.67:0/3809087916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6924102690 0x7f6924102a70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f690c0099c0 tx=0x7f690c009cd0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.249+0000 7f692a9af700 1 -- 172.21.15.67:0/3809087916 shutdown_connections 2024-05-03T17:22:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.249+0000 7f692a9af700 1 --2- 172.21.15.67:0/3809087916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f69241076e0 0x7f692410f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.249+0000 7f692a9af700 1 --2- 172.21.15.67:0/3809087916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6924102690 0x7f6924102a70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.249+0000 7f692a9af700 1 --2- 172.21.15.67:0/3809087916 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6924103040 0x7f6924107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.249+0000 7f692a9af700 1 -- 172.21.15.67:0/3809087916 >> 172.21.15.67:0/3809087916 conn(0x7f69240783f0 msgr2=0x7f6924078800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.249+0000 7f692a9af700 1 -- 172.21.15.67:0/3809087916 shutdown_connections 2024-05-03T17:22:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.249+0000 7f692a9af700 1 -- 172.21.15.67:0/3809087916 wait complete. 2024-05-03T17:22:30.261 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.249+0000 7f692a9af700 1 Processor -- start 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.249+0000 7f692a9af700 1 -- start start 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f692a9af700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6924102690 0x7f692419f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f692a9af700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6924103040 0x7f692419fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f692a9af700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69241076e0 0x7f6924199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f692a9af700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6924112570 con 0x7f6924102690 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f692a9af700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f69241123f0 con 0x7f6924103040 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f692a9af700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f69241126f0 con 0x7f69241076e0 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f6923fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6924102690 0x7f692419f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f6923fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6924102690 0x7f692419f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50780/0 (socket says 172.21.15.67:50780) 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f6923fff700 1 -- 172.21.15.67:0/1007374225 learned_addr learned my addr 172.21.15.67:0/1007374225 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f69237fe700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6924103040 0x7f692419fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f6928f4c700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69241076e0 0x7f6924199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f6923fff700 1 -- 172.21.15.67:0/1007374225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69241076e0 msgr2=0x7f6924199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f6923fff700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69241076e0 0x7f6924199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.262 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f6923fff700 1 -- 172.21.15.67:0/1007374225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6924103040 msgr2=0x7f692419fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f6923fff700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6924103040 0x7f692419fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f6923fff700 1 -- 172.21.15.67:0/1007374225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692419a1a0 con 0x7f6924102690 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f69237fe700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6924103040 0x7f692419fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f6928f4c700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69241076e0 0x7f6924199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f6923fff700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6924102690 0x7f692419f880 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f690c00b580 tx=0x7f690c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.250+0000 7f69217fa700 1 -- 172.21.15.67:0/1007374225 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c027070 con 0x7f6924102690 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.251+0000 7f692a9af700 1 -- 172.21.15.67:0/1007374225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f692419a430 con 0x7f6924102690 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.251+0000 7f69217fa700 1 -- 172.21.15.67:0/1007374225 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f690c004020 con 0x7f6924102690 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.251+0000 7f69217fa700 1 -- 172.21.15.67:0/1007374225 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f690c022e30 con 0x7f6924102690 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.251+0000 7f692a9af700 1 -- 172.21.15.67:0/1007374225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69241a6750 con 0x7f6924102690 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.251+0000 7f692a9af700 1 -- 172.21.15.67:0/1007374225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f692406a190 con 0x7f6924102690 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.253+0000 7f69217fa700 1 -- 172.21.15.67:0/1007374225 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f690c022590 con 0x7f6924102690 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.253+0000 7f69217fa700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f69100b2910 0x7f69100b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.254+0000 7f69237fe700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f69100b2910 0x7f69100b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.254+0000 7f69217fa700 1 -- 172.21.15.67:0/1007374225 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f690c0e34b0 con 0x7f6924102690 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.254+0000 7f69237fe700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f69100b2910 0x7f69100b4dd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6914005fd0 tx=0x7f6914005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:30.263 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.257+0000 7f69217fa700 1 -- 172.21.15.67:0/1007374225 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f690c030070 con 0x7f6924102690 2024-05-03T17:22:30.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:30 smithi070 ceph-mon[33250]: pgmap v352: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:22:30.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.475+0000 7f692a9af700 1 -- 172.21.15.67:0/1007374225 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f692402e510 con 0x7f6924102690 2024-05-03T17:22:30.476 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f69217fa700 1 -- 172.21.15.67:0/1007374225 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f690c013070 con 0x7f6924102690 2024-05-03T17:22:30.476 INFO:teuthology.orchestra.run.smithi067.stdout:55834574939 2024-05-03T17:22:30.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.479+0000 7f692a9af700 1 -- 172.21.15.67:0/1007374225 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f69100b2910 msgr2=0x7f69100b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:30.479 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.479+0000 7f692a9af700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f69100b2910 0x7f69100b4dd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6914005fd0 tx=0x7f6914005dc0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.479+0000 7f692a9af700 1 -- 172.21.15.67:0/1007374225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6924102690 msgr2=0x7f692419f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:30.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.479+0000 7f692a9af700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6924102690 0x7f692419f880 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f690c00b580 tx=0x7f690c0047b0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.479+0000 7f692a9af700 1 -- 172.21.15.67:0/1007374225 shutdown_connections 2024-05-03T17:22:30.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.479+0000 7f692a9af700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f69100b2910 0x7f69100b4dd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.479+0000 7f692a9af700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f69241076e0 0x7f6924199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.479+0000 7f692a9af700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6924102690 0x7f692419f880 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.479+0000 7f692a9af700 1 --2- 172.21.15.67:0/1007374225 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6924103040 0x7f692419fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.479+0000 7f692a9af700 1 -- 172.21.15.67:0/1007374225 >> 172.21.15.67:0/1007374225 conn(0x7f69240783f0 msgr2=0x7f692406c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:30.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.479+0000 7f692a9af700 1 -- 172.21.15.67:0/1007374225 shutdown_connections 2024-05-03T17:22:30.480 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.479+0000 7f692a9af700 1 -- 172.21.15.67:0/1007374225 wait complete. 2024-05-03T17:22:30.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.474+0000 7f8961f11700 1 -- 172.21.15.67:0/3932441511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c06bb40 msgr2=0x7f895c110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:30.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.474+0000 7f8961f11700 1 --2- 172.21.15.67:0/3932441511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c06bb40 0x7f895c110330 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f895400b0a0 tx=0x7f895400b3b0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.475+0000 7f8961f11700 1 -- 172.21.15.67:0/3932441511 shutdown_connections 2024-05-03T17:22:30.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.475+0000 7f8961f11700 1 --2- 172.21.15.67:0/3932441511 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c06bb40 0x7f895c110330 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.475+0000 7f8961f11700 1 --2- 172.21.15.67:0/3932441511 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f895c109510 0x7f895c1098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.475+0000 7f8961f11700 1 --2- 172.21.15.67:0/3932441511 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f895c06b180 0x7f895c06b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.488 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.475+0000 7f8961f11700 1 -- 172.21.15.67:0/3932441511 >> 172.21.15.67:0/3932441511 conn(0x7f895c079350 msgr2=0x7f895c079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.475+0000 7f8961f11700 1 -- 172.21.15.67:0/3932441511 shutdown_connections 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.475+0000 7f8961f11700 1 -- 172.21.15.67:0/3932441511 wait complete. 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.475+0000 7f8961f11700 1 Processor -- start 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f8961f11700 1 -- start start 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f8961f11700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f895c06b180 0x7f895c19f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f8961f11700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c06bb40 0x7f895c19fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f8961f11700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f895c109510 0x7f895c199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f8961f11700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f895c113640 con 0x7f895c06bb40 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f8961f11700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f895c1134c0 con 0x7f895c109510 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f8961f11700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f895c1137c0 con 0x7f895c06b180 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f895affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c06bb40 0x7f895c19fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f895affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c06bb40 0x7f895c19fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50798/0 (socket says 172.21.15.67:50798) 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f895affd700 1 -- 172.21.15.67:0/3576553129 learned_addr learned my addr 172.21.15.67:0/3576553129 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f895b7fe700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f895c06b180 0x7f895c19f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f895bfff700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f895c109510 0x7f895c199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f895affd700 1 -- 172.21.15.67:0/3576553129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f895c06b180 msgr2=0x7f895c19f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:30.489 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f895affd700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f895c06b180 0x7f895c19f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f895affd700 1 -- 172.21.15.67:0/3576553129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f895c109510 msgr2=0x7f895c199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f895affd700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f895c109510 0x7f895c199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f895affd700 1 -- 172.21.15.67:0/3576553129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895c19a1a0 con 0x7f895c06bb40 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f895b7fe700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f895c06b180 0x7f895c19f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.476+0000 7f895bfff700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f895c109510 0x7f895c199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.477+0000 7f895affd700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c06bb40 0x7f895c19fdc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f895000d950 tx=0x7f895000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.477+0000 7f8958ff9700 1 -- 172.21.15.67:0/3576553129 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950016070 con 0x7f895c06bb40 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.477+0000 7f8958ff9700 1 -- 172.21.15.67:0/3576553129 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8950010460 con 0x7f895c06bb40 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.477+0000 7f8961f11700 1 -- 172.21.15.67:0/3576553129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f895c19a490 con 0x7f895c06bb40 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.477+0000 7f8958ff9700 1 -- 172.21.15.67:0/3576553129 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f895000f620 con 0x7f895c06bb40 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.477+0000 7f8961f11700 1 -- 172.21.15.67:0/3576553129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f895c1a6750 con 0x7f895c06bb40 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.478+0000 7f8961f11700 1 -- 172.21.15.67:0/3576553129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f895c10b9f0 con 0x7f895c06bb40 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.480+0000 7f8958ff9700 1 -- 172.21.15.67:0/3576553129 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f8950019070 con 0x7f895c06bb40 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.481+0000 7f8958ff9700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f89440b2910 0x7f89440b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:30.490 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.481+0000 7f895b7fe700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f89440b2910 0x7f89440b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.481+0000 7f8958ff9700 1 -- 172.21.15.67:0/3576553129 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f89500e2020 con 0x7f895c06bb40 2024-05-03T17:22:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.481+0000 7f895b7fe700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f89440b2910 0x7f89440b4dd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f895c19ab80 tx=0x7f894c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:30.491 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.485+0000 7f8958ff9700 1 -- 172.21.15.67:0/3576553129 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f89500a9bd0 con 0x7f895c06bb40 2024-05-03T17:22:30.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:30 smithi082 ceph-mon[33535]: pgmap v352: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:22:30.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.702+0000 7f8961f11700 1 -- 172.21.15.67:0/3576553129 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f895c19af70 con 0x7f895c06bb40 2024-05-03T17:22:30.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.702+0000 7f8958ff9700 1 -- 172.21.15.67:0/3576553129 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f89500a94d0 con 0x7f895c06bb40 2024-05-03T17:22:30.703 INFO:teuthology.orchestra.run.smithi067.stdout:206158430267 2024-05-03T17:22:30.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.704+0000 7f8961f11700 1 -- 172.21.15.67:0/3576553129 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f89440b2910 msgr2=0x7f89440b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:30.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.704+0000 7f8961f11700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f89440b2910 0x7f89440b4dd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f895c19ab80 tx=0x7f894c009260 comp rx=0 tx=0).stop 2024-05-03T17:22:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.704+0000 7f8961f11700 1 -- 172.21.15.67:0/3576553129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c06bb40 msgr2=0x7f895c19fdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.704+0000 7f8961f11700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c06bb40 0x7f895c19fdc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f895000d950 tx=0x7f895000dd10 comp rx=0 tx=0).stop 2024-05-03T17:22:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.705+0000 7f8961f11700 1 -- 172.21.15.67:0/3576553129 shutdown_connections 2024-05-03T17:22:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.705+0000 7f8961f11700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f89440b2910 0x7f89440b4dd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.705+0000 7f8961f11700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f895c109510 0x7f895c199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.705+0000 7f8961f11700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f895c06b180 0x7f895c19f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.705+0000 7f8961f11700 1 --2- 172.21.15.67:0/3576553129 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f895c06bb40 0x7f895c19fdc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.705+0000 7f8961f11700 1 -- 172.21.15.67:0/3576553129 >> 172.21.15.67:0/3576553129 conn(0x7f895c079350 msgr2=0x7f895c101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.705+0000 7f8961f11700 1 -- 172.21.15.67:0/3576553129 shutdown_connections 2024-05-03T17:22:30.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:30.705+0000 7f8961f11700 1 -- 172.21.15.67:0/3576553129 wait complete. 2024-05-03T17:22:31.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:31 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1007374225' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-03T17:22:31.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:31 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3576553129' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-03T17:22:31.353 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:31 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1007374225' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-03T17:22:31.353 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:31 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3576553129' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-03T17:22:31.385 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574933 got 55834574939 for osd.1 2024-05-03T17:22:31.385 DEBUG:teuthology.parallel:result is None 2024-05-03T17:22:31.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:31 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1007374225' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-03T17:22:31.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:31 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3576553129' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-03T17:22:31.823 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:32.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:32 smithi067 ceph-mon[29731]: pgmap v353: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:22:32.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:32 smithi082 ceph-mon[33535]: pgmap v353: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:22:32.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:32 smithi070 ceph-mon[33250]: pgmap v353: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:22:32.759 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430259 got 206158430267 for osd.7 2024-05-03T17:22:32.759 DEBUG:teuthology.parallel:result is None 2024-05-03T17:22:33.495 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.520+0000 7f87b7864700 1 -- 172.21.15.67:0/3950095251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b0102d60 msgr2=0x7f87b010f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.520+0000 7f87b7864700 1 --2- 172.21.15.67:0/3950095251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b0102d60 0x7f87b010f250 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f87a800b0a0 tx=0x7f87a800b3b0 comp rx=0 tx=0).stop 2024-05-03T17:22:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.520+0000 7f87b7864700 1 -- 172.21.15.67:0/3950095251 shutdown_connections 2024-05-03T17:22:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.520+0000 7f87b7864700 1 --2- 172.21.15.67:0/3950095251 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b0102d60 0x7f87b010f250 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.520+0000 7f87b7864700 1 --2- 172.21.15.67:0/3950095251 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87b0108410 0x7f87b01087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.520+0000 7f87b7864700 1 --2- 172.21.15.67:0/3950095251 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87b01023a0 0x7f87b0102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.520+0000 7f87b7864700 1 -- 172.21.15.67:0/3950095251 >> 172.21.15.67:0/3950095251 conn(0x7f87b0079350 msgr2=0x7f87b0079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:33.521 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.520+0000 7f87b7864700 1 -- 172.21.15.67:0/3950095251 shutdown_connections 2024-05-03T17:22:33.522 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.520+0000 7f87b7864700 1 -- 172.21.15.67:0/3950095251 wait complete. 2024-05-03T17:22:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.521+0000 7f87b7864700 1 Processor -- start 2024-05-03T17:22:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.521+0000 7f87b7864700 1 -- start start 2024-05-03T17:22:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b7864700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87b01023a0 0x7f87b01a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b7864700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b0102d60 0x7f87b01a2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b7864700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87b0108410 0x7f87b019be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b7864700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f87b0112560 con 0x7f87b0102d60 2024-05-03T17:22:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b7864700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f87b01123e0 con 0x7f87b0108410 2024-05-03T17:22:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b7864700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f87b01126e0 con 0x7f87b01023a0 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b4dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b0102d60 0x7f87b01a2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b4dff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b0102d60 0x7f87b01a2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50810/0 (socket says 172.21.15.67:50810) 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b4dff700 1 -- 172.21.15.67:0/1644246158 learned_addr learned my addr 172.21.15.67:0/1644246158 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b5600700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87b01023a0 0x7f87b01a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b5e01700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87b0108410 0x7f87b019be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b4dff700 1 -- 172.21.15.67:0/1644246158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87b01023a0 msgr2=0x7f87b01a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b4dff700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87b01023a0 0x7f87b01a1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b4dff700 1 -- 172.21.15.67:0/1644246158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87b0108410 msgr2=0x7f87b019be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b4dff700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87b0108410 0x7f87b019be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.522+0000 7f87b4dff700 1 -- 172.21.15.67:0/1644246158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b019c740 con 0x7f87b0102d60 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.523+0000 7f87b4dff700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b0102d60 0x7f87b01a2000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f87a000d980 tx=0x7f87a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.523+0000 7f87a67fc700 1 -- 172.21.15.67:0/1644246158 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a0017070 con 0x7f87b0102d60 2024-05-03T17:22:33.525 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.523+0000 7f87b7864700 1 -- 172.21.15.67:0/1644246158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87b019ca30 con 0x7f87b0102d60 2024-05-03T17:22:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.523+0000 7f87b7864700 1 -- 172.21.15.67:0/1644246158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87b01a89d0 con 0x7f87b0102d60 2024-05-03T17:22:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.523+0000 7f87a67fc700 1 -- 172.21.15.67:0/1644246158 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87a0005020 con 0x7f87b0102d60 2024-05-03T17:22:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.523+0000 7f87a67fc700 1 -- 172.21.15.67:0/1644246158 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a000fa10 con 0x7f87b0102d60 2024-05-03T17:22:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.524+0000 7f87b7864700 1 -- 172.21.15.67:0/1644246158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87b0103ac0 con 0x7f87b0102d60 2024-05-03T17:22:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.526+0000 7f87a67fc700 1 -- 172.21.15.67:0/1644246158 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f87a0004a30 con 0x7f87b0102d60 2024-05-03T17:22:33.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.527+0000 7f87a67fc700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f879c0b2a10 0x7f879c0b4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:33.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.528+0000 7f87a67fc700 1 -- 172.21.15.67:0/1644246158 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f87a00e2640 con 0x7f87b0102d60 2024-05-03T17:22:33.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.528+0000 7f87b5600700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f879c0b2a10 0x7f879c0b4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:33.528 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.528+0000 7f87b5600700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f879c0b2a10 0x7f879c0b4ed0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f87ac005950 tx=0x7f87ac00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:33.531 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.530+0000 7f87a67fc700 1 -- 172.21.15.67:0/1644246158 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f87a0020020 con 0x7f87b0102d60 2024-05-03T17:22:33.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.749+0000 7f87b7864700 1 -- 172.21.15.67:0/1644246158 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f87b01a8c00 con 0x7f87b0102d60 2024-05-03T17:22:33.750 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.750+0000 7f87a67fc700 1 -- 172.21.15.67:0/1644246158 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f87a00aa210 con 0x7f87b0102d60 2024-05-03T17:22:33.750 INFO:teuthology.orchestra.run.smithi067.stdout:279172874284 2024-05-03T17:22:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.752+0000 7f87b7864700 1 -- 172.21.15.67:0/1644246158 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f879c0b2a10 msgr2=0x7f879c0b4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.752+0000 7f87b7864700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f879c0b2a10 0x7f879c0b4ed0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f87ac005950 tx=0x7f87ac00b410 comp rx=0 tx=0).stop 2024-05-03T17:22:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.752+0000 7f87b7864700 1 -- 172.21.15.67:0/1644246158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b0102d60 msgr2=0x7f87b01a2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.752+0000 7f87b7864700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b0102d60 0x7f87b01a2000 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f87a000d980 tx=0x7f87a000dc90 comp rx=0 tx=0).stop 2024-05-03T17:22:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.752+0000 7f87b7864700 1 -- 172.21.15.67:0/1644246158 shutdown_connections 2024-05-03T17:22:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.752+0000 7f87b7864700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f879c0b2a10 0x7f879c0b4ed0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.752+0000 7f87b7864700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f87b0108410 0x7f87b019be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.752+0000 7f87b7864700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f87b01023a0 0x7f87b01a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.752+0000 7f87b7864700 1 --2- 172.21.15.67:0/1644246158 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f87b0102d60 0x7f87b01a2000 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.752+0000 7f87b7864700 1 -- 172.21.15.67:0/1644246158 >> 172.21.15.67:0/1644246158 conn(0x7f87b0079350 msgr2=0x7f87b00ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.752+0000 7f87b7864700 1 -- 172.21.15.67:0/1644246158 shutdown_connections 2024-05-03T17:22:33.753 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:33.752+0000 7f87b7864700 1 -- 172.21.15.67:0/1644246158 wait complete. 2024-05-03T17:22:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.215+0000 7f674fd5e700 1 -- 172.21.15.67:0/641842939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6748108410 msgr2=0x7f67481087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.215+0000 7f674fd5e700 1 --2- 172.21.15.67:0/641842939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6748108410 0x7f67481087f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6738009990 tx=0x7f6738009ca0 comp rx=0 tx=0).stop 2024-05-03T17:22:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.216+0000 7f674fd5e700 1 -- 172.21.15.67:0/641842939 shutdown_connections 2024-05-03T17:22:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.216+0000 7f674fd5e700 1 --2- 172.21.15.67:0/641842939 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6748102d60 0x7f674810f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.216+0000 7f674fd5e700 1 --2- 172.21.15.67:0/641842939 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6748108410 0x7f67481087f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.216+0000 7f674fd5e700 1 --2- 172.21.15.67:0/641842939 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f67481023a0 0x7f6748102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.216+0000 7f674fd5e700 1 -- 172.21.15.67:0/641842939 >> 172.21.15.67:0/641842939 conn(0x7f6748079350 msgr2=0x7f6748079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:34.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.216+0000 7f674fd5e700 1 -- 172.21.15.67:0/641842939 shutdown_connections 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.216+0000 7f674fd5e700 1 -- 172.21.15.67:0/641842939 wait complete. 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.216+0000 7f674fd5e700 1 Processor -- start 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.216+0000 7f674fd5e700 1 -- start start 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.216+0000 7f674fd5e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67481023a0 0x7f674819f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.216+0000 7f674fd5e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6748102d60 0x7f674819fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674fd5e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6748108410 0x7f67481999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674fd5e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6748112560 con 0x7f67481023a0 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674fd5e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f67481123e0 con 0x7f6748102d60 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674fd5e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f67481126e0 con 0x7f6748108410 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674dafa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67481023a0 0x7f674819f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674dafa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67481023a0 0x7f674819f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50842/0 (socket says 172.21.15.67:50842) 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674dafa700 1 -- 172.21.15.67:0/2599510518 learned_addr learned my addr 172.21.15.67:0/2599510518 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674e2fb700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6748108410 0x7f67481999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:34.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674d2f9700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6748102d60 0x7f674819fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674dafa700 1 -- 172.21.15.67:0/2599510518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6748108410 msgr2=0x7f67481999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674dafa700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6748108410 0x7f67481999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674dafa700 1 -- 172.21.15.67:0/2599510518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6748102d60 msgr2=0x7f674819fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674dafa700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6748102d60 0x7f674819fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674dafa700 1 -- 172.21.15.67:0/2599510518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674819a100 con 0x7f67481023a0 2024-05-03T17:22:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674dafa700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67481023a0 0x7f674819f7e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6738000c00 tx=0x7f6738009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f673effd700 1 -- 172.21.15.67:0/2599510518 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6738027070 con 0x7f67481023a0 2024-05-03T17:22:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f673effd700 1 -- 172.21.15.67:0/2599510518 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6738022470 con 0x7f67481023a0 2024-05-03T17:22:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674fd5e700 1 -- 172.21.15.67:0/2599510518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f674819a390 con 0x7f67481023a0 2024-05-03T17:22:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f673effd700 1 -- 172.21.15.67:0/2599510518 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6738017650 con 0x7f67481023a0 2024-05-03T17:22:34.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.217+0000 7f674fd5e700 1 -- 172.21.15.67:0/2599510518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67481a66b0 con 0x7f67481023a0 2024-05-03T17:22:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.218+0000 7f674fd5e700 1 -- 172.21.15.67:0/2599510518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f674806a190 con 0x7f67481023a0 2024-05-03T17:22:34.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.221+0000 7f673effd700 1 -- 172.21.15.67:0/2599510518 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f673802a070 con 0x7f67481023a0 2024-05-03T17:22:34.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.221+0000 7f673effd700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f67340b2990 0x7f67340b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:34.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.222+0000 7f674d2f9700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f67340b2990 0x7f67340b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:34.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.222+0000 7f673effd700 1 -- 172.21.15.67:0/2599510518 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f67380e35f0 con 0x7f67481023a0 2024-05-03T17:22:34.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.222+0000 7f674d2f9700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f67340b2990 0x7f67340b4e50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6748079950 tx=0x7f6744009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:34.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.225+0000 7f673effd700 1 -- 172.21.15.67:0/2599510518 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6738013070 con 0x7f67481023a0 2024-05-03T17:22:34.338 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:34 smithi067 ceph-mon[29731]: pgmap v354: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:22:34.338 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:34 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1644246158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-03T17:22:34.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.443+0000 7f674fd5e700 1 -- 172.21.15.67:0/2599510518 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f67481a6880 con 0x7f67481023a0 2024-05-03T17:22:34.444 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.444+0000 7f673effd700 1 -- 172.21.15.67:0/2599510518 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f67380ab230 con 0x7f67481023a0 2024-05-03T17:22:34.444 INFO:teuthology.orchestra.run.smithi067.stdout:137438953548 2024-05-03T17:22:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.446+0000 7f674fd5e700 1 -- 172.21.15.67:0/2599510518 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f67340b2990 msgr2=0x7f67340b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.446+0000 7f674fd5e700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f67340b2990 0x7f67340b4e50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f6748079950 tx=0x7f6744009260 comp rx=0 tx=0).stop 2024-05-03T17:22:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.446+0000 7f674fd5e700 1 -- 172.21.15.67:0/2599510518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67481023a0 msgr2=0x7f674819f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.446+0000 7f674fd5e700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67481023a0 0x7f674819f7e0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6738000c00 tx=0x7f6738009580 comp rx=0 tx=0).stop 2024-05-03T17:22:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.446+0000 7f674fd5e700 1 -- 172.21.15.67:0/2599510518 shutdown_connections 2024-05-03T17:22:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.446+0000 7f674fd5e700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f67340b2990 0x7f67340b4e50 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.446+0000 7f674fd5e700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6748108410 0x7f67481999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.446+0000 7f674fd5e700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f67481023a0 0x7f674819f7e0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:34.447 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.446+0000 7f674fd5e700 1 --2- 172.21.15.67:0/2599510518 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6748102d60 0x7f674819fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:34.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.446+0000 7f674fd5e700 1 -- 172.21.15.67:0/2599510518 >> 172.21.15.67:0/2599510518 conn(0x7f6748079350 msgr2=0x7f674806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:34.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.446+0000 7f674fd5e700 1 -- 172.21.15.67:0/2599510518 shutdown_connections 2024-05-03T17:22:34.448 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:34.446+0000 7f674fd5e700 1 -- 172.21.15.67:0/2599510518 wait complete. 2024-05-03T17:22:34.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:34 smithi082 ceph-mon[33535]: pgmap v354: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:22:34.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:34 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1644246158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-03T17:22:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:34 smithi070 ceph-mon[33250]: pgmap v354: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:22:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:34 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1644246158' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-03T17:22:35.188 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874280 got 279172874284 for osd.10 2024-05-03T17:22:35.188 DEBUG:teuthology.parallel:result is None 2024-05-03T17:22:35.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:35 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2599510518' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-03T17:22:35.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:35 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2599510518' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-03T17:22:35.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:35 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2599510518' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-03T17:22:35.891 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953543 got 137438953548 for osd.4 2024-05-03T17:22:35.891 DEBUG:teuthology.parallel:result is None 2024-05-03T17:22:36.118 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:36 smithi067 ceph-mon[29731]: pgmap v355: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:22:36.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:36 smithi082 ceph-mon[33535]: pgmap v355: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:22:36.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:36 smithi070 ceph-mon[33250]: pgmap v355: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:22:37.338 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:37 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:22:37.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:37 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:22:37.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:37 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:22:37.561 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.561+0000 7f9caf09f700 1 -- 172.21.15.67:0/1607944274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ca8103750 msgr2=0x7f9ca8103b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:37.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.561+0000 7f9caf09f700 1 --2- 172.21.15.67:0/1607944274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ca8103750 0x7f9ca8103b30 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c0099c0 tx=0x7f9c9c009cd0 comp rx=0 tx=0).stop 2024-05-03T17:22:37.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.561+0000 7f9caf09f700 1 -- 172.21.15.67:0/1607944274 shutdown_connections 2024-05-03T17:22:37.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.561+0000 7f9caf09f700 1 --2- 172.21.15.67:0/1607944274 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ca8108850 0x7f9ca81103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:37.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.561+0000 7f9caf09f700 1 --2- 172.21.15.67:0/1607944274 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ca8103750 0x7f9ca8103b30 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:37.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.561+0000 7f9caf09f700 1 --2- 172.21.15.67:0/1607944274 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ca8104100 0x7f9ca8108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:37.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.561+0000 7f9caf09f700 1 -- 172.21.15.67:0/1607944274 >> 172.21.15.67:0/1607944274 conn(0x7f9ca807a600 msgr2=0x7f9ca807aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:37.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.561+0000 7f9caf09f700 1 -- 172.21.15.67:0/1607944274 shutdown_connections 2024-05-03T17:22:37.562 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.561+0000 7f9caf09f700 1 -- 172.21.15.67:0/1607944274 wait complete. 2024-05-03T17:22:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9caf09f700 1 Processor -- start 2024-05-03T17:22:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9caf09f700 1 -- start start 2024-05-03T17:22:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9caf09f700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ca8103750 0x7f9ca80757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9caf09f700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ca8104100 0x7f9ca8075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9caf09f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ca8108850 0x7f9ca8076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9caf09f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca8113640 con 0x7f9ca8108850 2024-05-03T17:22:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9caf09f700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca81134c0 con 0x7f9ca8103750 2024-05-03T17:22:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9caf09f700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca81137c0 con 0x7f9ca8104100 2024-05-03T17:22:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9cad63c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ca8108850 0x7f9ca8076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9cad63c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ca8108850 0x7f9ca8076220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59014/0 (socket says 172.21.15.67:59014) 2024-05-03T17:22:37.564 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9cad63c700 1 -- 172.21.15.67:0/3589251713 learned_addr learned my addr 172.21.15.67:0/3589251713 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9ca7fff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ca8104100 0x7f9ca8075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9cace3b700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ca8103750 0x7f9ca80757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9cad63c700 1 -- 172.21.15.67:0/3589251713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ca8104100 msgr2=0x7f9ca8075ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9cad63c700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ca8104100 0x7f9ca8075ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9cad63c700 1 -- 172.21.15.67:0/3589251713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ca8103750 msgr2=0x7f9ca80757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9cad63c700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ca8103750 0x7f9ca80757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.562+0000 7f9cad63c700 1 -- 172.21.15.67:0/3589251713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca81a8010 con 0x7f9ca8108850 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.563+0000 7f9cad63c700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ca8108850 0x7f9ca8076220 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000c370 tx=0x7f9ca000c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.563+0000 7f9ca5ffb700 1 -- 172.21.15.67:0/3589251713 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca00092e0 con 0x7f9ca8108850 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.563+0000 7f9caf09f700 1 -- 172.21.15.67:0/3589251713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca81a82a0 con 0x7f9ca8108850 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.563+0000 7f9ca5ffb700 1 -- 172.21.15.67:0/3589251713 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9ca0003a00 con 0x7f9ca8108850 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.563+0000 7f9ca5ffb700 1 -- 172.21.15.67:0/3589251713 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca00046f0 con 0x7f9ca8108850 2024-05-03T17:22:37.565 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.563+0000 7f9caf09f700 1 -- 172.21.15.67:0/3589251713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca81a87e0 con 0x7f9ca8108850 2024-05-03T17:22:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.564+0000 7f9caf09f700 1 -- 172.21.15.67:0/3589251713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca810ba90 con 0x7f9ca8108850 2024-05-03T17:22:37.566 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.565+0000 7f9ca5ffb700 1 -- 172.21.15.67:0/3589251713 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9ca001a020 con 0x7f9ca8108850 2024-05-03T17:22:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.567+0000 7f9ca5ffb700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9c980b2910 0x7f9c980b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.567+0000 7f9cace3b700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9c980b2910 0x7f9c980b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:37.567 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.567+0000 7f9ca5ffb700 1 -- 172.21.15.67:0/3589251713 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f9ca00e1bd0 con 0x7f9ca8108850 2024-05-03T17:22:37.568 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.567+0000 7f9cace3b700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9c980b2910 0x7f9c980b4dd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c003760 tx=0x7f9c9c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:37.572 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.572+0000 7f9ca5ffb700 1 -- 172.21.15.67:0/3589251713 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9ca0010040 con 0x7f9ca8108850 2024-05-03T17:22:37.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.797+0000 7f9caf09f700 1 -- 172.21.15.67:0/3589251713 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f9ca810b470 con 0x7f9ca8108850 2024-05-03T17:22:37.798 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.798+0000 7f9ca5ffb700 1 -- 172.21.15.67:0/3589251713 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9ca00a9780 con 0x7f9ca8108850 2024-05-03T17:22:37.799 INFO:teuthology.orchestra.run.smithi067.stdout:158913790024 2024-05-03T17:22:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.800+0000 7f9caf09f700 1 -- 172.21.15.67:0/3589251713 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9c980b2910 msgr2=0x7f9c980b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.800+0000 7f9caf09f700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9c980b2910 0x7f9c980b4dd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c003760 tx=0x7f9c9c005fb0 comp rx=0 tx=0).stop 2024-05-03T17:22:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.800+0000 7f9caf09f700 1 -- 172.21.15.67:0/3589251713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ca8108850 msgr2=0x7f9ca8076220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.800+0000 7f9caf09f700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ca8108850 0x7f9ca8076220 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9ca000c370 tx=0x7f9ca000c680 comp rx=0 tx=0).stop 2024-05-03T17:22:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.800+0000 7f9caf09f700 1 -- 172.21.15.67:0/3589251713 shutdown_connections 2024-05-03T17:22:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.800+0000 7f9caf09f700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9c980b2910 0x7f9c980b4dd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.800+0000 7f9caf09f700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9ca8108850 0x7f9ca8076220 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.800+0000 7f9caf09f700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9ca8103750 0x7f9ca80757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.800+0000 7f9caf09f700 1 --2- 172.21.15.67:0/3589251713 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9ca8104100 0x7f9ca8075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:37.801 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.800+0000 7f9caf09f700 1 -- 172.21.15.67:0/3589251713 >> 172.21.15.67:0/3589251713 conn(0x7f9ca807a600 msgr2=0x7f9ca81017b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:37.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.800+0000 7f9caf09f700 1 -- 172.21.15.67:0/3589251713 shutdown_connections 2024-05-03T17:22:37.802 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:37.800+0000 7f9caf09f700 1 -- 172.21.15.67:0/3589251713 wait complete. 2024-05-03T17:22:38.144 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:38 smithi082 ceph-mon[33535]: pgmap v356: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:38.145 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:38 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3589251713' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-03T17:22:38.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:38 smithi070 ceph-mon[33250]: pgmap v356: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:38.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:38 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3589251713' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-03T17:22:38.354 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:38 smithi067 ceph-mon[29731]: pgmap v356: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:38.354 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:38 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3589251713' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-03T17:22:38.712 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790019 got 158913790024 for osd.5 2024-05-03T17:22:38.712 DEBUG:teuthology.parallel:result is None 2024-05-03T17:22:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.169+0000 7f19df648700 1 -- 172.21.15.67:0/752587630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d806c3d0 msgr2=0x7f19d8110320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.169+0000 7f19df648700 1 --2- 172.21.15.67:0/752587630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d806c3d0 0x7f19d8110320 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f19d000b0a0 tx=0x7f19d000b3b0 comp rx=0 tx=0).stop 2024-05-03T17:22:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.169+0000 7f19df648700 1 -- 172.21.15.67:0/752587630 shutdown_connections 2024-05-03T17:22:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.169+0000 7f19df648700 1 --2- 172.21.15.67:0/752587630 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d806c3d0 0x7f19d8110320 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.169+0000 7f19df648700 1 --2- 172.21.15.67:0/752587630 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19d8109500 0x7f19d81098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.169+0000 7f19df648700 1 --2- 172.21.15.67:0/752587630 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19d806ba10 0x7f19d806be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.169+0000 7f19df648700 1 -- 172.21.15.67:0/752587630 >> 172.21.15.67:0/752587630 conn(0x7f19d8079310 msgr2=0x7f19d8079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.169+0000 7f19df648700 1 -- 172.21.15.67:0/752587630 shutdown_connections 2024-05-03T17:22:39.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.169+0000 7f19df648700 1 -- 172.21.15.67:0/752587630 wait complete. 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.169+0000 7f19df648700 1 Processor -- start 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.169+0000 7f19df648700 1 -- start start 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19df648700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19d806ba10 0x7f19d819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19df648700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d806c3d0 0x7f19d819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19df648700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19d8109500 0x7f19d8199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19df648700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f19d8113630 con 0x7f19d806c3d0 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19df648700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f19d81134b0 con 0x7f19d8109500 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19df648700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f19d81137b0 con 0x7f19d806ba10 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19dcbe3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d806c3d0 0x7f19d819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19dd3e4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19d806ba10 0x7f19d819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19dd3e4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19d806ba10 0x7f19d819f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:44414/0 (socket says 172.21.15.67:44414) 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19dd3e4700 1 -- 172.21.15.67:0/2615633781 learned_addr learned my addr 172.21.15.67:0/2615633781 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19dcbe3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d806c3d0 0x7f19d819fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59034/0 (socket says 172.21.15.67:59034) 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19ddbe5700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19d8109500 0x7f19d8199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:39.173 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19dd3e4700 1 -- 172.21.15.67:0/2615633781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19d8109500 msgr2=0x7f19d8199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19dd3e4700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19d8109500 0x7f19d8199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19dd3e4700 1 -- 172.21.15.67:0/2615633781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d806c3d0 msgr2=0x7f19d819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19dd3e4700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d806c3d0 0x7f19d819fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19dd3e4700 1 -- 172.21.15.67:0/2615633781 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19d819a070 con 0x7f19d806ba10 2024-05-03T17:22:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19ddbe5700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19d8109500 0x7f19d8199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:22:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.170+0000 7f19dd3e4700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19d806ba10 0x7f19d819f750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f19c800e9a0 tx=0x7f19c800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.171+0000 7f19ce7fc700 1 -- 172.21.15.67:0/2615633781 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c800ca10 con 0x7f19d806ba10 2024-05-03T17:22:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.171+0000 7f19ce7fc700 1 -- 172.21.15.67:0/2615633781 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19c800cbb0 con 0x7f19d806ba10 2024-05-03T17:22:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.171+0000 7f19df648700 1 -- 172.21.15.67:0/2615633781 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19d819a360 con 0x7f19d806ba10 2024-05-03T17:22:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.171+0000 7f19ce7fc700 1 -- 172.21.15.67:0/2615633781 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c8018680 con 0x7f19d806ba10 2024-05-03T17:22:39.174 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.171+0000 7f19df648700 1 -- 172.21.15.67:0/2615633781 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19d8102390 con 0x7f19d806ba10 2024-05-03T17:22:39.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.172+0000 7f19df648700 1 -- 172.21.15.67:0/2615633781 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19d806a190 con 0x7f19d806ba10 2024-05-03T17:22:39.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.173+0000 7f19ce7fc700 1 -- 172.21.15.67:0/2615633781 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f19c8010430 con 0x7f19d806ba10 2024-05-03T17:22:39.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.174+0000 7f19ce7fc700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f19c40b2910 0x7f19c40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:39.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.174+0000 7f19dcbe3700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f19c40b2910 0x7f19c40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:39.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.174+0000 7f19ce7fc700 1 -- 172.21.15.67:0/2615633781 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f19c80e24c0 con 0x7f19d806ba10 2024-05-03T17:22:39.175 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.174+0000 7f19dcbe3700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f19c40b2910 0x7f19c40b4dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f19d8079980 tx=0x7f19d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:39.179 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.179+0000 7f19ce7fc700 1 -- 172.21.15.67:0/2615633781 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f19c80aa070 con 0x7f19d806ba10 2024-05-03T17:22:39.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.406+0000 7f19df648700 1 -- 172.21.15.67:0/2615633781 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f19d802e510 con 0x7f19d806ba10 2024-05-03T17:22:39.407 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.406+0000 7f19ce7fc700 1 -- 172.21.15.67:0/2615633781 <== mon.1 v2:172.21.15.82:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f19c80b20f0 con 0x7f19d806ba10 2024-05-03T17:22:39.407 INFO:teuthology.orchestra.run.smithi067.stdout:77309411416 2024-05-03T17:22:39.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.409+0000 7f19df648700 1 -- 172.21.15.67:0/2615633781 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f19c40b2910 msgr2=0x7f19c40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:39.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.409+0000 7f19df648700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f19c40b2910 0x7f19c40b4dd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f19d8079980 tx=0x7f19d4009260 comp rx=0 tx=0).stop 2024-05-03T17:22:39.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.409+0000 7f19df648700 1 -- 172.21.15.67:0/2615633781 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19d806ba10 msgr2=0x7f19d819f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:39.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.409+0000 7f19df648700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19d806ba10 0x7f19d819f750 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f19c800e9a0 tx=0x7f19c800ecb0 comp rx=0 tx=0).stop 2024-05-03T17:22:39.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.409+0000 7f19df648700 1 -- 172.21.15.67:0/2615633781 shutdown_connections 2024-05-03T17:22:39.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.409+0000 7f19df648700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f19c40b2910 0x7f19c40b4dd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:39.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.409+0000 7f19df648700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19d8109500 0x7f19d8199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:39.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.409+0000 7f19df648700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19d806ba10 0x7f19d819f750 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:39.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.409+0000 7f19df648700 1 --2- 172.21.15.67:0/2615633781 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19d806c3d0 0x7f19d819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:39.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.409+0000 7f19df648700 1 -- 172.21.15.67:0/2615633781 >> 172.21.15.67:0/2615633781 conn(0x7f19d8079310 msgr2=0x7f19d8100f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:39.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.409+0000 7f19df648700 1 -- 172.21.15.67:0/2615633781 shutdown_connections 2024-05-03T17:22:39.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:39.409+0000 7f19df648700 1 -- 172.21.15.67:0/2615633781 wait complete. 2024-05-03T17:22:40.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:39 smithi070 ceph-mon[33250]: pgmap v357: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-03T17:22:40.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:40.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:40.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:40.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:39 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:40.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:39 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2615633781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-03T17:22:40.159 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:39 smithi067 ceph-mon[29731]: pgmap v357: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-03T17:22:40.159 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:40.159 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:40.159 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:40.159 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:39 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:40.159 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:39 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2615633781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-03T17:22:40.185 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411411 got 77309411416 for osd.2 2024-05-03T17:22:40.185 DEBUG:teuthology.parallel:result is None 2024-05-03T17:22:40.185 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-03T17:22:40.185 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-03T17:22:40.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:39 smithi082 ceph-mon[33535]: pgmap v357: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-03T17:22:40.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:40.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:40.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:40.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:39 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:40.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:39 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2615633781' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-03T17:22:40.669 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:42.046 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:41 smithi067 ceph-mon[29731]: pgmap v358: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:22:42.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:41 smithi070 ceph-mon[33250]: pgmap v358: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:22:42.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:41 smithi082 ceph-mon[33535]: pgmap v358: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:22:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.410+0000 7feabedd8700 1 -- 172.21.15.67:0/2560372399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab8102620 msgr2=0x7feab8102a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.410+0000 7feabedd8700 1 --2- 172.21.15.67:0/2560372399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab8102620 0x7feab8102a00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7feaac0099c0 tx=0x7feaac009cd0 comp rx=0 tx=0).stop 2024-05-03T17:22:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.411+0000 7feabedd8700 1 -- 172.21.15.67:0/2560372399 shutdown_connections 2024-05-03T17:22:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.411+0000 7feabedd8700 1 --2- 172.21.15.67:0/2560372399 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab8107700 0x7feab810f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.411+0000 7feabedd8700 1 --2- 172.21.15.67:0/2560372399 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab8102620 0x7feab8102a00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.411+0000 7feabedd8700 1 --2- 172.21.15.67:0/2560372399 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feab8102fd0 0x7feab8107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.411+0000 7feabedd8700 1 -- 172.21.15.67:0/2560372399 >> 172.21.15.67:0/2560372399 conn(0x7feab8079440 msgr2=0x7feab8079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.411+0000 7feabedd8700 1 -- 172.21.15.67:0/2560372399 shutdown_connections 2024-05-03T17:22:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.411+0000 7feabedd8700 1 -- 172.21.15.67:0/2560372399 wait complete. 2024-05-03T17:22:42.412 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.411+0000 7feabedd8700 1 Processor -- start 2024-05-03T17:22:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.411+0000 7feabedd8700 1 -- start start 2024-05-03T17:22:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabedd8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab8102620 0x7feab819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabedd8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feab8102fd0 0x7feab819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabedd8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab8107700 0x7feab81998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabedd8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7feab8112590 con 0x7feab8102620 2024-05-03T17:22:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabedd8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7feab8112410 con 0x7feab8102fd0 2024-05-03T17:22:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabedd8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7feab8112710 con 0x7feab8107700 2024-05-03T17:22:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabcb74700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab8102620 0x7feab819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabcb74700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab8102620 0x7feab819f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59042/0 (socket says 172.21.15.67:59042) 2024-05-03T17:22:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabcb74700 1 -- 172.21.15.67:0/469975098 learned_addr learned my addr 172.21.15.67:0/469975098 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:42.414 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feab7fff700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feab8102fd0 0x7feab819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabd375700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab8107700 0x7feab81998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabcb74700 1 -- 172.21.15.67:0/469975098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab8107700 msgr2=0x7feab81998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabcb74700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab8107700 0x7feab81998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabcb74700 1 -- 172.21.15.67:0/469975098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feab8102fd0 msgr2=0x7feab819fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabcb74700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feab8102fd0 0x7feab819fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabcb74700 1 -- 172.21.15.67:0/469975098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab819a040 con 0x7feab8102620 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feab7fff700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feab8102fd0 0x7feab819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.412+0000 7feabcb74700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab8102620 0x7feab819f720 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7feaac0098e0 tx=0x7feaac0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.413+0000 7feab5ffb700 1 -- 172.21.15.67:0/469975098 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac027070 con 0x7feab8102620 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.413+0000 7feabedd8700 1 -- 172.21.15.67:0/469975098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab819a2d0 con 0x7feab8102620 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.413+0000 7feab5ffb700 1 -- 172.21.15.67:0/469975098 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feaac004020 con 0x7feab8102620 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.413+0000 7feab5ffb700 1 -- 172.21.15.67:0/469975098 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaac022e10 con 0x7feab8102620 2024-05-03T17:22:42.415 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.413+0000 7feabedd8700 1 -- 172.21.15.67:0/469975098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feab806cec0 con 0x7feab8102620 2024-05-03T17:22:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.414+0000 7feabedd8700 1 -- 172.21.15.67:0/469975098 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab8103b00 con 0x7feab8102620 2024-05-03T17:22:42.416 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.416+0000 7feab5ffb700 1 -- 172.21.15.67:0/469975098 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7feaac02a070 con 0x7feab8102620 2024-05-03T17:22:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.416+0000 7feab5ffb700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7feaa80b2910 0x7feaa80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.416+0000 7feab7fff700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7feaa80b2910 0x7feaa80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.416+0000 7feab5ffb700 1 -- 172.21.15.67:0/469975098 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7feaac005030 con 0x7feab8102620 2024-05-03T17:22:42.417 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.417+0000 7feab7fff700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7feaa80b2910 0x7feaa80b4dd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7feab8108c80 tx=0x7feaa400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:42.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.424+0000 7feab5ffb700 1 -- 172.21.15.67:0/469975098 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7feaac031020 con 0x7feab8102620 2024-05-03T17:22:42.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.663+0000 7feabedd8700 1 -- 172.21.15.67:0/469975098 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feab819ae10 con 0x7feaa80b2910 2024-05-03T17:22:42.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.671+0000 7feab5ffb700 1 -- 172.21.15.67:0/469975098 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258920 (secure 0 0 0) 0x7feab819ae10 con 0x7feaa80b2910 2024-05-03T17:22:42.675 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:22:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.674+0000 7feabedd8700 1 -- 172.21.15.67:0/469975098 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7feaa80b2910 msgr2=0x7feaa80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.674+0000 7feabedd8700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7feaa80b2910 0x7feaa80b4dd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7feab8108c80 tx=0x7feaa400a380 comp rx=0 tx=0).stop 2024-05-03T17:22:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.674+0000 7feabedd8700 1 -- 172.21.15.67:0/469975098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab8102620 msgr2=0x7feab819f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.674+0000 7feabedd8700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab8102620 0x7feab819f720 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7feaac0098e0 tx=0x7feaac0057c0 comp rx=0 tx=0).stop 2024-05-03T17:22:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.674+0000 7feabedd8700 1 -- 172.21.15.67:0/469975098 shutdown_connections 2024-05-03T17:22:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.674+0000 7feabedd8700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7feaa80b2910 0x7feaa80b4dd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.674+0000 7feabedd8700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7feab8107700 0x7feab81998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.674+0000 7feabedd8700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7feab8102620 0x7feab819f720 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.674+0000 7feabedd8700 1 --2- 172.21.15.67:0/469975098 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7feab8102fd0 0x7feab819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.674+0000 7feabedd8700 1 -- 172.21.15.67:0/469975098 >> 172.21.15.67:0/469975098 conn(0x7feab8079440 msgr2=0x7feab806beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.674+0000 7feabedd8700 1 -- 172.21.15.67:0/469975098 shutdown_connections 2024-05-03T17:22:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:42.674+0000 7feabedd8700 1 -- 172.21.15.67:0/469975098 wait complete. 2024-05-03T17:22:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-03T17:22:43.433 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":359,"stamp":"2024-05-03T17:22:42.447880+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":646436,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":702,"num_read_kb":433,"num_write":287,"num_write_kb":2254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":231,"ondisk_log_size":231,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":359904,"kb_used_data":10728,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581920,"statfs":{"total":1151940427776,"available":1151571886080,"internally_reserved":0,"allocated":10985472,"data_stored":7071828,"data_compressed":178158,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":65,"num_read_kb":30,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002873"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595780+0000","last_change":"2024-05-03T17:20:37.204469+0000","last_active":"2024-05-03T17:21:07.595780+0000","last_peered":"2024-05-03T17:21:07.595780+0000","last_clean":"2024-05-03T17:21:07.595780+0000","last_became_active":"2024-05-03T17:20:37.204098+0000","last_became_peered":"2024-05-03T17:20:37.204098+0000","last_unstale":"2024-05-03T17:21:07.595780+0000","last_undegraded":"2024-05-03T17:21:07.595780+0000","last_fullsized":"2024-05-03T17:21:07.595780+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:03:29.875319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599098+0000","last_change":"2024-05-03T17:20:37.201171+0000","last_active":"2024-05-03T17:21:07.599098+0000","last_peered":"2024-05-03T17:21:07.599098+0000","last_clean":"2024-05-03T17:21:07.599098+0000","last_became_active":"2024-05-03T17:20:37.200852+0000","last_became_peered":"2024-05-03T17:20:37.200852+0000","last_unstale":"2024-05-03T17:21:07.599098+0000","last_undegraded":"2024-05-03T17:21:07.599098+0000","last_fullsized":"2024-05-03T17:21:07.599098+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:12:06.549919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598922+0000","last_change":"2024-05-03T17:20:37.201925+0000","last_active":"2024-05-03T17:21:07.598922+0000","last_peered":"2024-05-03T17:21:07.598922+0000","last_clean":"2024-05-03T17:21:07.598922+0000","last_became_active":"2024-05-03T17:20:37.201619+0000","last_became_peered":"2024-05-03T17:20:37.201619+0000","last_unstale":"2024-05-03T17:21:07.598922+0000","last_undegraded":"2024-05-03T17:21:07.598922+0000","last_fullsized":"2024-05-03T17:21:07.598922+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:50:19.019887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603188+0000","last_change":"2024-05-03T17:20:37.203535+0000","last_active":"2024-05-03T17:21:07.603188+0000","last_peered":"2024-05-03T17:21:07.603188+0000","last_clean":"2024-05-03T17:21:07.603188+0000","last_became_active":"2024-05-03T17:20:37.203175+0000","last_became_peered":"2024-05-03T17:20:37.203175+0000","last_unstale":"2024-05-03T17:21:07.603188+0000","last_undegraded":"2024-05-03T17:21:07.603188+0000","last_fullsized":"2024-05-03T17:21:07.603188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:59:41.026709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597618+0000","last_change":"2024-05-03T17:20:37.200283+0000","last_active":"2024-05-03T17:21:07.597618+0000","last_peered":"2024-05-03T17:21:07.597618+0000","last_clean":"2024-05-03T17:21:07.597618+0000","last_became_active":"2024-05-03T17:20:37.199105+0000","last_became_peered":"2024-05-03T17:20:37.199105+0000","last_unstale":"2024-05-03T17:21:07.597618+0000","last_undegraded":"2024-05-03T17:21:07.597618+0000","last_fullsized":"2024-05-03T17:21:07.597618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:56:39.990497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597382+0000","last_change":"2024-05-03T17:21:07.597382+0000","last_active":"2024-05-03T17:21:07.597382+0000","last_peered":"2024-05-03T17:21:07.597382+0000","last_clean":"2024-05-03T17:21:07.597382+0000","last_became_active":"2024-05-03T17:21:07.596935+0000","last_became_peered":"2024-05-03T17:21:07.596935+0000","last_unstale":"2024-05-03T17:21:07.597382+0000","last_undegraded":"2024-05-03T17:21:07.597382+0000","last_fullsized":"2024-05-03T17:21:07.597382+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T19:30:50.200783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596543+0000","last_change":"2024-05-03T17:20:37.211378+0000","last_active":"2024-05-03T17:21:07.596543+0000","last_peered":"2024-05-03T17:21:07.596543+0000","last_clean":"2024-05-03T17:21:07.596543+0000","last_became_active":"2024-05-03T17:20:37.210684+0000","last_became_peered":"2024-05-03T17:20:37.210684+0000","last_unstale":"2024-05-03T17:21:07.596543+0000","last_undegraded":"2024-05-03T17:21:07.596543+0000","last_fullsized":"2024-05-03T17:21:07.596543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:35:24.737186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592032+0000","last_change":"2024-05-03T17:20:37.202989+0000","last_active":"2024-05-03T17:21:07.592032+0000","last_peered":"2024-05-03T17:21:07.592032+0000","last_clean":"2024-05-03T17:21:07.592032+0000","last_became_active":"2024-05-03T17:20:37.199975+0000","last_became_peered":"2024-05-03T17:20:37.199975+0000","last_unstale":"2024-05-03T17:21:07.592032+0000","last_undegraded":"2024-05-03T17:21:07.592032+0000","last_fullsized":"2024-05-03T17:21:07.592032+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:13:43.875472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598772+0000","last_change":"2024-05-03T17:20:37.206754+0000","last_active":"2024-05-03T17:21:07.598772+0000","last_peered":"2024-05-03T17:21:07.598772+0000","last_clean":"2024-05-03T17:21:07.598772+0000","last_became_active":"2024-05-03T17:20:37.206423+0000","last_became_peered":"2024-05-03T17:20:37.206423+0000","last_unstale":"2024-05-03T17:21:07.598772+0000","last_undegraded":"2024-05-03T17:21:07.598772+0000","last_fullsized":"2024-05-03T17:21:07.598772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:10:30.227969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593231+0000","last_change":"2024-05-03T17:20:37.211868+0000","last_active":"2024-05-03T17:21:07.593231+0000","last_peered":"2024-05-03T17:21:07.593231+0000","last_clean":"2024-05-03T17:21:07.593231+0000","last_became_active":"2024-05-03T17:20:37.211486+0000","last_became_peered":"2024-05-03T17:20:37.211486+0000","last_unstale":"2024-05-03T17:21:07.593231+0000","last_undegraded":"2024-05-03T17:21:07.593231+0000","last_fullsized":"2024-05-03T17:21:07.593231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:41:03.966446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596847+0000","last_change":"2024-05-03T17:20:37.206511+0000","last_active":"2024-05-03T17:21:07.596847+0000","last_peered":"2024-05-03T17:21:07.596847+0000","last_clean":"2024-05-03T17:21:07.596847+0000","last_became_active":"2024-05-03T17:20:37.206180+0000","last_became_peered":"2024-05-03T17:20:37.206180+0000","last_unstale":"2024-05-03T17:21:07.596847+0000","last_undegraded":"2024-05-03T17:21:07.596847+0000","last_fullsized":"2024-05-03T17:21:07.596847+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:59:07.277623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603068+0000","last_change":"2024-05-03T17:20:37.204842+0000","last_active":"2024-05-03T17:21:07.603068+0000","last_peered":"2024-05-03T17:21:07.603068+0000","last_clean":"2024-05-03T17:21:07.603068+0000","last_became_active":"2024-05-03T17:20:37.204275+0000","last_became_peered":"2024-05-03T17:20:37.204275+0000","last_unstale":"2024-05-03T17:21:07.603068+0000","last_undegraded":"2024-05-03T17:21:07.603068+0000","last_fullsized":"2024-05-03T17:21:07.603068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:27:00.021413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597215+0000","last_change":"2024-05-03T17:21:07.597215+0000","last_active":"2024-05-03T17:21:07.597215+0000","last_peered":"2024-05-03T17:21:07.597215+0000","last_clean":"2024-05-03T17:21:07.597215+0000","last_became_active":"2024-05-03T17:21:07.596833+0000","last_became_peered":"2024-05-03T17:21:07.596833+0000","last_unstale":"2024-05-03T17:21:07.597215+0000","last_undegraded":"2024-05-03T17:21:07.597215+0000","last_fullsized":"2024-05-03T17:21:07.597215+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:09:41.399761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597704+0000","last_change":"2024-05-03T17:20:37.200759+0000","last_active":"2024-05-03T17:21:07.597704+0000","last_peered":"2024-05-03T17:21:07.597704+0000","last_clean":"2024-05-03T17:21:07.597704+0000","last_became_active":"2024-05-03T17:20:37.200410+0000","last_became_peered":"2024-05-03T17:20:37.200410+0000","last_unstale":"2024-05-03T17:21:07.597704+0000","last_undegraded":"2024-05-03T17:21:07.597704+0000","last_fullsized":"2024-05-03T17:21:07.597704+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:51:03.719792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597446+0000","last_change":"2024-05-03T17:20:37.204534+0000","last_active":"2024-05-03T17:21:07.597446+0000","last_peered":"2024-05-03T17:21:07.597446+0000","last_clean":"2024-05-03T17:21:07.597446+0000","last_became_active":"2024-05-03T17:20:37.204167+0000","last_became_peered":"2024-05-03T17:20:37.204167+0000","last_unstale":"2024-05-03T17:21:07.597446+0000","last_undegraded":"2024-05-03T17:21:07.597446+0000","last_fullsized":"2024-05-03T17:21:07.597446+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:31:35.730465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598463+0000","last_change":"2024-05-03T17:20:37.207658+0000","last_active":"2024-05-03T17:21:07.598463+0000","last_peered":"2024-05-03T17:21:07.598463+0000","last_clean":"2024-05-03T17:21:07.598463+0000","last_became_active":"2024-05-03T17:20:37.207313+0000","last_became_peered":"2024-05-03T17:20:37.207313+0000","last_unstale":"2024-05-03T17:21:07.598463+0000","last_undegraded":"2024-05-03T17:21:07.598463+0000","last_fullsized":"2024-05-03T17:21:07.598463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:47:21.287559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595876+0000","last_change":"2024-05-03T17:20:37.205021+0000","last_active":"2024-05-03T17:21:07.595876+0000","last_peered":"2024-05-03T17:21:07.595876+0000","last_clean":"2024-05-03T17:21:07.595876+0000","last_became_active":"2024-05-03T17:20:37.204630+0000","last_became_peered":"2024-05-03T17:20:37.204630+0000","last_unstale":"2024-05-03T17:21:07.595876+0000","last_undegraded":"2024-05-03T17:21:07.595876+0000","last_fullsized":"2024-05-03T17:21:07.595876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:03:00.318648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593860+0000","last_change":"2024-05-03T17:20:37.201829+0000","last_active":"2024-05-03T17:21:07.593860+0000","last_peered":"2024-05-03T17:21:07.593860+0000","last_clean":"2024-05-03T17:21:07.593860+0000","last_became_active":"2024-05-03T17:20:37.201465+0000","last_became_peered":"2024-05-03T17:20:37.201465+0000","last_unstale":"2024-05-03T17:21:07.593860+0000","last_undegraded":"2024-05-03T17:21:07.593860+0000","last_fullsized":"2024-05-03T17:21:07.593860+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:05:46.844150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594023+0000","last_change":"2024-05-03T17:20:37.199138+0000","last_active":"2024-05-03T17:21:07.594023+0000","last_peered":"2024-05-03T17:21:07.594023+0000","last_clean":"2024-05-03T17:21:07.594023+0000","last_became_active":"2024-05-03T17:20:37.198722+0000","last_became_peered":"2024-05-03T17:20:37.198722+0000","last_unstale":"2024-05-03T17:21:07.594023+0000","last_undegraded":"2024-05-03T17:21:07.594023+0000","last_fullsized":"2024-05-03T17:21:07.594023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:12:04.764356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594031+0000","last_change":"2024-05-03T17:20:33.168111+0000","last_active":"2024-05-03T17:21:07.594031+0000","last_peered":"2024-05-03T17:21:07.594031+0000","last_clean":"2024-05-03T17:21:07.594031+0000","last_became_active":"2024-05-03T17:20:33.167892+0000","last_became_peered":"2024-05-03T17:20:33.167892+0000","last_unstale":"2024-05-03T17:21:07.594031+0000","last_undegraded":"2024-05-03T17:21:07.594031+0000","last_fullsized":"2024-05-03T17:21:07.594031+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:55:45.235287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592267+0000","last_change":"2024-05-03T17:20:37.208673+0000","last_active":"2024-05-03T17:21:07.592267+0000","last_peered":"2024-05-03T17:21:07.592267+0000","last_clean":"2024-05-03T17:21:07.592267+0000","last_became_active":"2024-05-03T17:20:37.208360+0000","last_became_peered":"2024-05-03T17:20:37.208360+0000","last_unstale":"2024-05-03T17:21:07.592267+0000","last_undegraded":"2024-05-03T17:21:07.592267+0000","last_fullsized":"2024-05-03T17:21:07.592267+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:42:02.966225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592646+0000","last_change":"2024-05-03T17:20:33.284052+0000","last_active":"2024-05-03T17:21:07.592646+0000","last_peered":"2024-05-03T17:21:07.592646+0000","last_clean":"2024-05-03T17:21:07.592646+0000","last_became_active":"2024-05-03T17:20:33.283635+0000","last_became_peered":"2024-05-03T17:20:33.283635+0000","last_unstale":"2024-05-03T17:21:07.592646+0000","last_undegraded":"2024-05-03T17:21:07.592646+0000","last_fullsized":"2024-05-03T17:21:07.592646+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:17:51.219359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598804+0000","last_change":"2024-05-03T17:20:37.197321+0000","last_active":"2024-05-03T17:21:07.598804+0000","last_peered":"2024-05-03T17:21:07.598804+0000","last_clean":"2024-05-03T17:21:07.598804+0000","last_became_active":"2024-05-03T17:20:37.196569+0000","last_became_peered":"2024-05-03T17:20:37.196569+0000","last_unstale":"2024-05-03T17:21:07.598804+0000","last_undegraded":"2024-05-03T17:21:07.598804+0000","last_fullsized":"2024-05-03T17:21:07.598804+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:38:03.274058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598373+0000","last_change":"2024-05-03T17:20:33.168828+0000","last_active":"2024-05-03T17:21:07.598373+0000","last_peered":"2024-05-03T17:21:07.598373+0000","last_clean":"2024-05-03T17:21:07.598373+0000","last_became_active":"2024-05-03T17:20:33.168408+0000","last_became_peered":"2024-05-03T17:20:33.168408+0000","last_unstale":"2024-05-03T17:21:07.598373+0000","last_undegraded":"2024-05-03T17:21:07.598373+0000","last_fullsized":"2024-05-03T17:21:07.598373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:50:57.790353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598587+0000","last_change":"2024-05-03T17:20:37.199853+0000","last_active":"2024-05-03T17:21:07.598587+0000","last_peered":"2024-05-03T17:21:07.598587+0000","last_clean":"2024-05-03T17:21:07.598587+0000","last_became_active":"2024-05-03T17:20:37.199526+0000","last_became_peered":"2024-05-03T17:20:37.199526+0000","last_unstale":"2024-05-03T17:21:07.598587+0000","last_undegraded":"2024-05-03T17:21:07.598587+0000","last_fullsized":"2024-05-03T17:21:07.598587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:42:45.932795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597857+0000","last_change":"2024-05-03T17:20:33.168050+0000","last_active":"2024-05-03T17:21:07.597857+0000","last_peered":"2024-05-03T17:21:07.597857+0000","last_clean":"2024-05-03T17:21:07.597857+0000","last_became_active":"2024-05-03T17:20:33.167479+0000","last_became_peered":"2024-05-03T17:20:33.167479+0000","last_unstale":"2024-05-03T17:21:07.597857+0000","last_undegraded":"2024-05-03T17:21:07.597857+0000","last_fullsized":"2024-05-03T17:21:07.597857+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:39:32.626940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604399+0000","last_change":"2024-05-03T17:21:07.603996+0000","last_active":"2024-05-03T17:21:07.604399+0000","last_peered":"2024-05-03T17:21:07.604399+0000","last_clean":"2024-05-03T17:21:07.604399+0000","last_became_active":"2024-05-03T17:21:07.603647+0000","last_became_peered":"2024-05-03T17:21:07.603647+0000","last_unstale":"2024-05-03T17:21:07.604399+0000","last_undegraded":"2024-05-03T17:21:07.604399+0000","last_fullsized":"2024-05-03T17:21:07.604399+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:31:07.956642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"79'6","reported_seq":139,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:22:40.731321+0000","last_change":"2024-05-03T17:20:33.166560+0000","last_active":"2024-05-03T17:22:40.731321+0000","last_peered":"2024-05-03T17:22:40.731321+0000","last_clean":"2024-05-03T17:22:40.731321+0000","last_became_active":"2024-05-03T17:20:33.166136+0000","last_became_peered":"2024-05-03T17:20:33.166136+0000","last_unstale":"2024-05-03T17:22:40.731321+0000","last_undegraded":"2024-05-03T17:22:40.731321+0000","last_fullsized":"2024-05-03T17:22:40.731321+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:38:54.132364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597525+0000","last_change":"2024-05-03T17:20:37.204336+0000","last_active":"2024-05-03T17:21:07.597525+0000","last_peered":"2024-05-03T17:21:07.597525+0000","last_clean":"2024-05-03T17:21:07.597525+0000","last_became_active":"2024-05-03T17:20:37.203962+0000","last_became_peered":"2024-05-03T17:20:37.203962+0000","last_unstale":"2024-05-03T17:21:07.597525+0000","last_undegraded":"2024-05-03T17:21:07.597525+0000","last_fullsized":"2024-05-03T17:21:07.597525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:29:48.081362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592586+0000","last_change":"2024-05-03T17:20:33.171046+0000","last_active":"2024-05-03T17:21:07.592586+0000","last_peered":"2024-05-03T17:21:07.592586+0000","last_clean":"2024-05-03T17:21:07.592586+0000","last_became_active":"2024-05-03T17:20:33.169410+0000","last_became_peered":"2024-05-03T17:20:33.169410+0000","last_unstale":"2024-05-03T17:21:07.592586+0000","last_undegraded":"2024-05-03T17:21:07.592586+0000","last_fullsized":"2024-05-03T17:21:07.592586+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:45:05.987672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604346+0000","last_change":"2024-05-03T17:20:37.210695+0000","last_active":"2024-05-03T17:21:07.604346+0000","last_peered":"2024-05-03T17:21:07.604346+0000","last_clean":"2024-05-03T17:21:07.604346+0000","last_became_active":"2024-05-03T17:20:37.210285+0000","last_became_peered":"2024-05-03T17:20:37.210285+0000","last_unstale":"2024-05-03T17:21:07.604346+0000","last_undegraded":"2024-05-03T17:21:07.604346+0000","last_fullsized":"2024-05-03T17:21:07.604346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:10:52.013017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595054+0000","last_change":"2024-05-03T17:20:33.163997+0000","last_active":"2024-05-03T17:21:07.595054+0000","last_peered":"2024-05-03T17:21:07.595054+0000","last_clean":"2024-05-03T17:21:07.595054+0000","last_became_active":"2024-05-03T17:20:33.163653+0000","last_became_peered":"2024-05-03T17:20:33.163653+0000","last_unstale":"2024-05-03T17:21:07.595054+0000","last_undegraded":"2024-05-03T17:21:07.595054+0000","last_fullsized":"2024-05-03T17:21:07.595054+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:55:51.762133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592099+0000","last_change":"2024-05-03T17:20:37.195967+0000","last_active":"2024-05-03T17:21:07.592099+0000","last_peered":"2024-05-03T17:21:07.592099+0000","last_clean":"2024-05-03T17:21:07.592099+0000","last_became_active":"2024-05-03T17:20:37.195652+0000","last_became_peered":"2024-05-03T17:20:37.195652+0000","last_unstale":"2024-05-03T17:21:07.592099+0000","last_undegraded":"2024-05-03T17:21:07.592099+0000","last_fullsized":"2024-05-03T17:21:07.592099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:34:37.381525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603936+0000","last_change":"2024-05-03T17:20:33.167378+0000","last_active":"2024-05-03T17:21:07.603936+0000","last_peered":"2024-05-03T17:21:07.603936+0000","last_clean":"2024-05-03T17:21:07.603936+0000","last_became_active":"2024-05-03T17:20:33.167000+0000","last_became_peered":"2024-05-03T17:20:33.167000+0000","last_unstale":"2024-05-03T17:21:07.603936+0000","last_undegraded":"2024-05-03T17:21:07.603936+0000","last_fullsized":"2024-05-03T17:21:07.603936+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:53:48.087190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598616+0000","last_change":"2024-05-03T17:21:07.598616+0000","last_active":"2024-05-03T17:21:07.598616+0000","last_peered":"2024-05-03T17:21:07.598616+0000","last_clean":"2024-05-03T17:21:07.598616+0000","last_became_active":"2024-05-03T17:21:07.598234+0000","last_became_peered":"2024-05-03T17:21:07.598234+0000","last_unstale":"2024-05-03T17:21:07.598616+0000","last_undegraded":"2024-05-03T17:21:07.598616+0000","last_fullsized":"2024-05-03T17:21:07.598616+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:18:59.968590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'3","reported_seq":128,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:22:35.724529+0000","last_change":"2024-05-03T17:20:33.172079+0000","last_active":"2024-05-03T17:22:35.724529+0000","last_peered":"2024-05-03T17:22:35.724529+0000","last_clean":"2024-05-03T17:22:35.724529+0000","last_became_active":"2024-05-03T17:20:33.171323+0000","last_became_peered":"2024-05-03T17:20:33.171323+0000","last_unstale":"2024-05-03T17:22:35.724529+0000","last_undegraded":"2024-05-03T17:22:35.724529+0000","last_fullsized":"2024-05-03T17:22:35.724529+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:52:16.091897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":99,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599057+0000","last_change":"2024-05-03T17:20:37.202340+0000","last_active":"2024-05-03T17:21:07.599057+0000","last_peered":"2024-05-03T17:21:07.599057+0000","last_clean":"2024-05-03T17:21:07.599057+0000","last_became_active":"2024-05-03T17:20:37.201986+0000","last_became_peered":"2024-05-03T17:20:37.201986+0000","last_unstale":"2024-05-03T17:21:07.599057+0000","last_undegraded":"2024-05-03T17:21:07.599057+0000","last_fullsized":"2024-05-03T17:21:07.599057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:13:47.492653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.611413+0000","last_change":"2024-05-03T17:21:07.611413+0000","last_active":"2024-05-03T17:21:07.611413+0000","last_peered":"2024-05-03T17:21:07.611413+0000","last_clean":"2024-05-03T17:21:07.611413+0000","last_became_active":"2024-05-03T17:21:07.604448+0000","last_became_peered":"2024-05-03T17:21:07.604448+0000","last_unstale":"2024-05-03T17:21:07.611413+0000","last_undegraded":"2024-05-03T17:21:07.611413+0000","last_fullsized":"2024-05-03T17:21:07.611413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:28:29.639741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604785+0000","last_change":"2024-05-03T17:21:07.604785+0000","last_active":"2024-05-03T17:21:07.604785+0000","last_peered":"2024-05-03T17:21:07.604785+0000","last_clean":"2024-05-03T17:21:07.604785+0000","last_became_active":"2024-05-03T17:21:07.604507+0000","last_became_peered":"2024-05-03T17:21:07.604507+0000","last_unstale":"2024-05-03T17:21:07.604785+0000","last_undegraded":"2024-05-03T17:21:07.604785+0000","last_fullsized":"2024-05-03T17:21:07.604785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:53:52.501538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604562+0000","last_change":"2024-05-03T17:20:33.284585+0000","last_active":"2024-05-03T17:21:07.604562+0000","last_peered":"2024-05-03T17:21:07.604562+0000","last_clean":"2024-05-03T17:21:07.604562+0000","last_became_active":"2024-05-03T17:20:33.284203+0000","last_became_peered":"2024-05-03T17:20:33.284203+0000","last_unstale":"2024-05-03T17:21:07.604562+0000","last_undegraded":"2024-05-03T17:21:07.604562+0000","last_fullsized":"2024-05-03T17:21:07.604562+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:40:34.608382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598750+0000","last_change":"2024-05-03T17:20:37.203127+0000","last_active":"2024-05-03T17:21:07.598750+0000","last_peered":"2024-05-03T17:21:07.598750+0000","last_clean":"2024-05-03T17:21:07.598750+0000","last_became_active":"2024-05-03T17:20:37.202809+0000","last_became_peered":"2024-05-03T17:20:37.202809+0000","last_unstale":"2024-05-03T17:21:07.598750+0000","last_undegraded":"2024-05-03T17:21:07.598750+0000","last_fullsized":"2024-05-03T17:21:07.598750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:36:38.136107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'27","reported_seq":259,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:22:36.725945+0000","last_change":"2024-05-03T17:20:33.171487+0000","last_active":"2024-05-03T17:22:36.725945+0000","last_peered":"2024-05-03T17:22:36.725945+0000","last_clean":"2024-05-03T17:22:36.725945+0000","last_became_active":"2024-05-03T17:20:33.171021+0000","last_became_peered":"2024-05-03T17:20:33.171021+0000","last_unstale":"2024-05-03T17:22:36.725945+0000","last_undegraded":"2024-05-03T17:22:36.725945+0000","last_fullsized":"2024-05-03T17:22:36.725945+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:20:05.729189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51833,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":206,"num_read_kb":153,"num_write":28,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597839+0000","last_change":"2024-05-03T17:20:37.208443+0000","last_active":"2024-05-03T17:21:07.597839+0000","last_peered":"2024-05-03T17:21:07.597839+0000","last_clean":"2024-05-03T17:21:07.597839+0000","last_became_active":"2024-05-03T17:20:37.207869+0000","last_became_peered":"2024-05-03T17:20:37.207869+0000","last_unstale":"2024-05-03T17:21:07.597839+0000","last_undegraded":"2024-05-03T17:21:07.597839+0000","last_fullsized":"2024-05-03T17:21:07.597839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:13:28.978735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594065+0000","last_change":"2024-05-03T17:20:33.167368+0000","last_active":"2024-05-03T17:21:07.594065+0000","last_peered":"2024-05-03T17:21:07.594065+0000","last_clean":"2024-05-03T17:21:07.594065+0000","last_became_active":"2024-05-03T17:20:33.166503+0000","last_became_peered":"2024-05-03T17:20:33.166503+0000","last_unstale":"2024-05-03T17:21:07.594065+0000","last_undegraded":"2024-05-03T17:21:07.594065+0000","last_fullsized":"2024-05-03T17:21:07.594065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:14:23.150421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595589+0000","last_change":"2024-05-03T17:20:37.201887+0000","last_active":"2024-05-03T17:21:07.595589+0000","last_peered":"2024-05-03T17:21:07.595589+0000","last_clean":"2024-05-03T17:21:07.595589+0000","last_became_active":"2024-05-03T17:20:37.201548+0000","last_became_peered":"2024-05-03T17:20:37.201548+0000","last_unstale":"2024-05-03T17:21:07.595589+0000","last_undegraded":"2024-05-03T17:21:07.595589+0000","last_fullsized":"2024-05-03T17:21:07.595589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:39:26.416885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604320+0000","last_change":"2024-05-03T17:20:33.168039+0000","last_active":"2024-05-03T17:21:07.604320+0000","last_peered":"2024-05-03T17:21:07.604320+0000","last_clean":"2024-05-03T17:21:07.604320+0000","last_became_active":"2024-05-03T17:20:33.165695+0000","last_became_peered":"2024-05-03T17:20:33.165695+0000","last_unstale":"2024-05-03T17:21:07.604320+0000","last_undegraded":"2024-05-03T17:21:07.604320+0000","last_fullsized":"2024-05-03T17:21:07.604320+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:47:17.964849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596442+0000","last_change":"2024-05-03T17:20:37.207618+0000","last_active":"2024-05-03T17:21:07.596442+0000","last_peered":"2024-05-03T17:21:07.596442+0000","last_clean":"2024-05-03T17:21:07.596442+0000","last_became_active":"2024-05-03T17:20:37.207290+0000","last_became_peered":"2024-05-03T17:20:37.207290+0000","last_unstale":"2024-05-03T17:21:07.596442+0000","last_undegraded":"2024-05-03T17:21:07.596442+0000","last_fullsized":"2024-05-03T17:21:07.596442+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:26:34.085385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'3","reported_seq":131,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:22:36.726749+0000","last_change":"2024-05-03T17:20:33.167328+0000","last_active":"2024-05-03T17:22:36.726749+0000","last_peered":"2024-05-03T17:22:36.726749+0000","last_clean":"2024-05-03T17:22:36.726749+0000","last_became_active":"2024-05-03T17:20:33.166920+0000","last_became_peered":"2024-05-03T17:20:33.166920+0000","last_unstale":"2024-05-03T17:22:36.726749+0000","last_undegraded":"2024-05-03T17:22:36.726749+0000","last_fullsized":"2024-05-03T17:22:36.726749+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T19:04:47.450596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":103,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596215+0000","last_change":"2024-05-03T17:20:37.212691+0000","last_active":"2024-05-03T17:21:07.596215+0000","last_peered":"2024-05-03T17:21:07.596215+0000","last_clean":"2024-05-03T17:21:07.596215+0000","last_became_active":"2024-05-03T17:20:37.208692+0000","last_became_peered":"2024-05-03T17:20:37.208692+0000","last_unstale":"2024-05-03T17:21:07.596215+0000","last_undegraded":"2024-05-03T17:21:07.596215+0000","last_fullsized":"2024-05-03T17:21:07.596215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:04:29.441208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597477+0000","last_change":"2024-05-03T17:20:33.285266+0000","last_active":"2024-05-03T17:21:07.597477+0000","last_peered":"2024-05-03T17:21:07.597477+0000","last_clean":"2024-05-03T17:21:07.597477+0000","last_became_active":"2024-05-03T17:20:33.284223+0000","last_became_peered":"2024-05-03T17:20:33.284223+0000","last_unstale":"2024-05-03T17:21:07.597477+0000","last_undegraded":"2024-05-03T17:21:07.597477+0000","last_fullsized":"2024-05-03T17:21:07.597477+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:20:49.077832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604104+0000","last_change":"2024-05-03T17:20:37.205094+0000","last_active":"2024-05-03T17:21:07.604104+0000","last_peered":"2024-05-03T17:21:07.604104+0000","last_clean":"2024-05-03T17:21:07.604104+0000","last_became_active":"2024-05-03T17:20:37.204769+0000","last_became_peered":"2024-05-03T17:20:37.204769+0000","last_unstale":"2024-05-03T17:21:07.604104+0000","last_undegraded":"2024-05-03T17:21:07.604104+0000","last_fullsized":"2024-05-03T17:21:07.604104+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:00:32.129259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598498+0000","last_change":"2024-05-03T17:20:33.167920+0000","last_active":"2024-05-03T17:21:07.598498+0000","last_peered":"2024-05-03T17:21:07.598498+0000","last_clean":"2024-05-03T17:21:07.598498+0000","last_became_active":"2024-05-03T17:20:33.167571+0000","last_became_peered":"2024-05-03T17:20:33.167571+0000","last_unstale":"2024-05-03T17:21:07.598498+0000","last_undegraded":"2024-05-03T17:21:07.598498+0000","last_fullsized":"2024-05-03T17:21:07.598498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:13:23.462127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595627+0000","last_change":"2024-05-03T17:20:37.208051+0000","last_active":"2024-05-03T17:21:07.595627+0000","last_peered":"2024-05-03T17:21:07.595627+0000","last_clean":"2024-05-03T17:21:07.595627+0000","last_became_active":"2024-05-03T17:20:37.207685+0000","last_became_peered":"2024-05-03T17:20:37.207685+0000","last_unstale":"2024-05-03T17:21:07.595627+0000","last_undegraded":"2024-05-03T17:21:07.595627+0000","last_fullsized":"2024-05-03T17:21:07.595627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:27:47.685727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593733+0000","last_change":"2024-05-03T17:20:33.285337+0000","last_active":"2024-05-03T17:21:07.593733+0000","last_peered":"2024-05-03T17:21:07.593733+0000","last_clean":"2024-05-03T17:21:07.593733+0000","last_became_active":"2024-05-03T17:20:33.285012+0000","last_became_peered":"2024-05-03T17:20:33.285012+0000","last_unstale":"2024-05-03T17:21:07.593733+0000","last_undegraded":"2024-05-03T17:21:07.593733+0000","last_fullsized":"2024-05-03T17:21:07.593733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:04:44.246674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593083+0000","last_change":"2024-05-03T17:20:37.211216+0000","last_active":"2024-05-03T17:21:07.593083+0000","last_peered":"2024-05-03T17:21:07.593083+0000","last_clean":"2024-05-03T17:21:07.593083+0000","last_became_active":"2024-05-03T17:20:37.210906+0000","last_became_peered":"2024-05-03T17:20:37.210906+0000","last_unstale":"2024-05-03T17:21:07.593083+0000","last_undegraded":"2024-05-03T17:21:07.593083+0000","last_fullsized":"2024-05-03T17:21:07.593083+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:20:45.831133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597079+0000","last_change":"2024-05-03T17:20:33.165952+0000","last_active":"2024-05-03T17:21:07.597079+0000","last_peered":"2024-05-03T17:21:07.597079+0000","last_clean":"2024-05-03T17:21:07.597079+0000","last_became_active":"2024-05-03T17:20:33.165546+0000","last_became_peered":"2024-05-03T17:20:33.165546+0000","last_unstale":"2024-05-03T17:21:07.597079+0000","last_undegraded":"2024-05-03T17:21:07.597079+0000","last_fullsized":"2024-05-03T17:21:07.597079+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:26:44.484883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592776+0000","last_change":"2024-05-03T17:20:37.209107+0000","last_active":"2024-05-03T17:21:07.592776+0000","last_peered":"2024-05-03T17:21:07.592776+0000","last_clean":"2024-05-03T17:21:07.592776+0000","last_became_active":"2024-05-03T17:20:37.208754+0000","last_became_peered":"2024-05-03T17:20:37.208754+0000","last_unstale":"2024-05-03T17:21:07.592776+0000","last_undegraded":"2024-05-03T17:21:07.592776+0000","last_fullsized":"2024-05-03T17:21:07.592776+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:47:22.142530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604315+0000","last_change":"2024-05-03T17:20:37.209092+0000","last_active":"2024-05-03T17:21:07.604315+0000","last_peered":"2024-05-03T17:21:07.604315+0000","last_clean":"2024-05-03T17:21:07.604315+0000","last_became_active":"2024-05-03T17:20:37.208618+0000","last_became_peered":"2024-05-03T17:20:37.208618+0000","last_unstale":"2024-05-03T17:21:07.604315+0000","last_undegraded":"2024-05-03T17:21:07.604315+0000","last_fullsized":"2024-05-03T17:21:07.604315+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:11:09.509930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'4","reported_seq":132,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:22:38.728856+0000","last_change":"2024-05-03T17:20:33.283985+0000","last_active":"2024-05-03T17:22:38.728856+0000","last_peered":"2024-05-03T17:22:38.728856+0000","last_clean":"2024-05-03T17:22:38.728856+0000","last_became_active":"2024-05-03T17:20:33.283629+0000","last_became_peered":"2024-05-03T17:20:33.283629+0000","last_unstale":"2024-05-03T17:22:38.728856+0000","last_undegraded":"2024-05-03T17:22:38.728856+0000","last_fullsized":"2024-05-03T17:22:38.728856+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:03:08.751017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":101,"num_read_kb":101,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592649+0000","last_change":"2024-05-03T17:20:37.209086+0000","last_active":"2024-05-03T17:21:07.592649+0000","last_peered":"2024-05-03T17:21:07.592649+0000","last_clean":"2024-05-03T17:21:07.592649+0000","last_became_active":"2024-05-03T17:20:37.207968+0000","last_became_peered":"2024-05-03T17:20:37.207968+0000","last_unstale":"2024-05-03T17:21:07.592649+0000","last_undegraded":"2024-05-03T17:21:07.592649+0000","last_fullsized":"2024-05-03T17:21:07.592649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:57:51.816995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603078+0000","last_change":"2024-05-03T17:20:33.164302+0000","last_active":"2024-05-03T17:21:07.603078+0000","last_peered":"2024-05-03T17:21:07.603078+0000","last_clean":"2024-05-03T17:21:07.603078+0000","last_became_active":"2024-05-03T17:20:33.163958+0000","last_became_peered":"2024-05-03T17:20:33.163958+0000","last_unstale":"2024-05-03T17:21:07.603078+0000","last_undegraded":"2024-05-03T17:21:07.603078+0000","last_fullsized":"2024-05-03T17:21:07.603078+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:09:08.221659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":101,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596376+0000","last_change":"2024-05-03T17:19:04.188548+0000","last_active":"2024-05-03T17:21:07.596376+0000","last_peered":"2024-05-03T17:21:07.596376+0000","last_clean":"2024-05-03T17:21:07.596376+0000","last_became_active":"2024-05-03T17:19:04.183986+0000","last_became_peered":"2024-05-03T17:19:04.183986+0000","last_unstale":"2024-05-03T17:21:07.596376+0000","last_undegraded":"2024-05-03T17:21:07.596376+0000","last_fullsized":"2024-05-03T17:21:07.596376+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:15:37.276361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:15:37.276361+0000","last_clean_scrub_stamp":"2024-05-03T17:15:37.276361+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:45:57.109987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594820+0000","last_change":"2024-05-03T17:20:37.206686+0000","last_active":"2024-05-03T17:21:07.594820+0000","last_peered":"2024-05-03T17:21:07.594820+0000","last_clean":"2024-05-03T17:21:07.594820+0000","last_became_active":"2024-05-03T17:20:37.204810+0000","last_became_peered":"2024-05-03T17:20:37.204810+0000","last_unstale":"2024-05-03T17:21:07.594820+0000","last_undegraded":"2024-05-03T17:21:07.594820+0000","last_fullsized":"2024-05-03T17:21:07.594820+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:52:11.363372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597296+0000","last_change":"2024-05-03T17:20:33.168412+0000","last_active":"2024-05-03T17:21:07.597296+0000","last_peered":"2024-05-03T17:21:07.597296+0000","last_clean":"2024-05-03T17:21:07.597296+0000","last_became_active":"2024-05-03T17:20:33.168089+0000","last_became_peered":"2024-05-03T17:20:33.168089+0000","last_unstale":"2024-05-03T17:21:07.597296+0000","last_undegraded":"2024-05-03T17:21:07.597296+0000","last_fullsized":"2024-05-03T17:21:07.597296+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:03:11.983138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.602992+0000","last_change":"2024-05-03T17:21:07.602540+0000","last_active":"2024-05-03T17:21:07.602992+0000","last_peered":"2024-05-03T17:21:07.602992+0000","last_clean":"2024-05-03T17:21:07.602992+0000","last_became_active":"2024-05-03T17:21:07.599329+0000","last_became_peered":"2024-05-03T17:21:07.599329+0000","last_unstale":"2024-05-03T17:21:07.602992+0000","last_undegraded":"2024-05-03T17:21:07.602992+0000","last_fullsized":"2024-05-03T17:21:07.602992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:57:16.367541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594903+0000","last_change":"2024-05-03T17:20:37.206615+0000","last_active":"2024-05-03T17:21:07.594903+0000","last_peered":"2024-05-03T17:21:07.594903+0000","last_clean":"2024-05-03T17:21:07.594903+0000","last_became_active":"2024-05-03T17:20:37.205519+0000","last_became_peered":"2024-05-03T17:20:37.205519+0000","last_unstale":"2024-05-03T17:21:07.594903+0000","last_undegraded":"2024-05-03T17:21:07.594903+0000","last_fullsized":"2024-05-03T17:21:07.594903+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T19:08:45.459749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'11","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:21.216256+0000","last_change":"2024-05-03T17:20:33.164933+0000","last_active":"2024-05-03T17:21:21.216256+0000","last_peered":"2024-05-03T17:21:21.216256+0000","last_clean":"2024-05-03T17:21:21.216256+0000","last_became_active":"2024-05-03T17:20:33.164403+0000","last_became_peered":"2024-05-03T17:20:33.164403+0000","last_unstale":"2024-05-03T17:21:21.216256+0000","last_undegraded":"2024-05-03T17:21:21.216256+0000","last_fullsized":"2024-05-03T17:21:21.216256+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:11:21.158371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592505+0000","last_change":"2024-05-03T17:20:37.211145+0000","last_active":"2024-05-03T17:21:07.592505+0000","last_peered":"2024-05-03T17:21:07.592505+0000","last_clean":"2024-05-03T17:21:07.592505+0000","last_became_active":"2024-05-03T17:20:37.210829+0000","last_became_peered":"2024-05-03T17:20:37.210829+0000","last_unstale":"2024-05-03T17:21:07.592505+0000","last_undegraded":"2024-05-03T17:21:07.592505+0000","last_fullsized":"2024-05-03T17:21:07.592505+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:19:23.868001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603017+0000","last_change":"2024-05-03T17:20:33.166287+0000","last_active":"2024-05-03T17:21:07.603017+0000","last_peered":"2024-05-03T17:21:07.603017+0000","last_clean":"2024-05-03T17:21:07.603017+0000","last_became_active":"2024-05-03T17:20:33.165929+0000","last_became_peered":"2024-05-03T17:20:33.165929+0000","last_unstale":"2024-05-03T17:21:07.603017+0000","last_undegraded":"2024-05-03T17:21:07.603017+0000","last_fullsized":"2024-05-03T17:21:07.603017+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:32:37.093941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599190+0000","last_change":"2024-05-03T17:20:37.186668+0000","last_active":"2024-05-03T17:21:07.599190+0000","last_peered":"2024-05-03T17:21:07.599190+0000","last_clean":"2024-05-03T17:21:07.599190+0000","last_became_active":"2024-05-03T17:20:37.185922+0000","last_became_peered":"2024-05-03T17:20:37.185922+0000","last_unstale":"2024-05-03T17:21:07.599190+0000","last_undegraded":"2024-05-03T17:21:07.599190+0000","last_fullsized":"2024-05-03T17:21:07.599190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:00:11.075950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598468+0000","last_change":"2024-05-03T17:20:33.284515+0000","last_active":"2024-05-03T17:21:07.598468+0000","last_peered":"2024-05-03T17:21:07.598468+0000","last_clean":"2024-05-03T17:21:07.598468+0000","last_became_active":"2024-05-03T17:20:33.284169+0000","last_became_peered":"2024-05-03T17:20:33.284169+0000","last_unstale":"2024-05-03T17:21:07.598468+0000","last_undegraded":"2024-05-03T17:21:07.598468+0000","last_fullsized":"2024-05-03T17:21:07.598468+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:27:49.722159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597482+0000","last_change":"2024-05-03T17:20:37.200724+0000","last_active":"2024-05-03T17:21:07.597482+0000","last_peered":"2024-05-03T17:21:07.597482+0000","last_clean":"2024-05-03T17:21:07.597482+0000","last_became_active":"2024-05-03T17:20:37.200383+0000","last_became_peered":"2024-05-03T17:20:37.200383+0000","last_unstale":"2024-05-03T17:21:07.597482+0000","last_undegraded":"2024-05-03T17:21:07.597482+0000","last_fullsized":"2024-05-03T17:21:07.597482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:12:15.547433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592389+0000","last_change":"2024-05-03T17:20:33.163642+0000","last_active":"2024-05-03T17:21:07.592389+0000","last_peered":"2024-05-03T17:21:07.592389+0000","last_clean":"2024-05-03T17:21:07.592389+0000","last_became_active":"2024-05-03T17:20:33.163270+0000","last_became_peered":"2024-05-03T17:20:33.163270+0000","last_unstale":"2024-05-03T17:21:07.592389+0000","last_undegraded":"2024-05-03T17:21:07.592389+0000","last_fullsized":"2024-05-03T17:21:07.592389+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:48:02.719294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595341+0000","last_change":"2024-05-03T17:20:37.203063+0000","last_active":"2024-05-03T17:21:07.595341+0000","last_peered":"2024-05-03T17:21:07.595341+0000","last_clean":"2024-05-03T17:21:07.595341+0000","last_became_active":"2024-05-03T17:20:37.202609+0000","last_became_peered":"2024-05-03T17:20:37.202609+0000","last_unstale":"2024-05-03T17:21:07.595341+0000","last_undegraded":"2024-05-03T17:21:07.595341+0000","last_fullsized":"2024-05-03T17:21:07.595341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:38:08.497937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593609+0000","last_change":"2024-05-03T17:20:33.168672+0000","last_active":"2024-05-03T17:21:07.593609+0000","last_peered":"2024-05-03T17:21:07.593609+0000","last_clean":"2024-05-03T17:21:07.593609+0000","last_became_active":"2024-05-03T17:20:33.168349+0000","last_became_peered":"2024-05-03T17:20:33.168349+0000","last_unstale":"2024-05-03T17:21:07.593609+0000","last_undegraded":"2024-05-03T17:21:07.593609+0000","last_fullsized":"2024-05-03T17:21:07.593609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:26:04.340040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592378+0000","last_change":"2024-05-03T17:20:37.203725+0000","last_active":"2024-05-03T17:21:07.592378+0000","last_peered":"2024-05-03T17:21:07.592378+0000","last_clean":"2024-05-03T17:21:07.592378+0000","last_became_active":"2024-05-03T17:20:37.203318+0000","last_became_peered":"2024-05-03T17:20:37.203318+0000","last_unstale":"2024-05-03T17:21:07.592378+0000","last_undegraded":"2024-05-03T17:21:07.592378+0000","last_fullsized":"2024-05-03T17:21:07.592378+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:00:48.706036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.609823+0000","last_change":"2024-05-03T17:21:07.609392+0000","last_active":"2024-05-03T17:21:07.609823+0000","last_peered":"2024-05-03T17:21:07.609823+0000","last_clean":"2024-05-03T17:21:07.609823+0000","last_became_active":"2024-05-03T17:21:07.600420+0000","last_became_peered":"2024-05-03T17:21:07.600420+0000","last_unstale":"2024-05-03T17:21:07.609823+0000","last_undegraded":"2024-05-03T17:21:07.609823+0000","last_fullsized":"2024-05-03T17:21:07.609823+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:03:47.869058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597082+0000","last_change":"2024-05-03T17:20:37.203766+0000","last_active":"2024-05-03T17:21:07.597082+0000","last_peered":"2024-05-03T17:21:07.597082+0000","last_clean":"2024-05-03T17:21:07.597082+0000","last_became_active":"2024-05-03T17:20:37.203306+0000","last_became_peered":"2024-05-03T17:20:37.203306+0000","last_unstale":"2024-05-03T17:21:07.597082+0000","last_undegraded":"2024-05-03T17:21:07.597082+0000","last_fullsized":"2024-05-03T17:21:07.597082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:57:22.760234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594059+0000","last_change":"2024-05-03T17:20:33.285014+0000","last_active":"2024-05-03T17:21:07.594059+0000","last_peered":"2024-05-03T17:21:07.594059+0000","last_clean":"2024-05-03T17:21:07.594059+0000","last_became_active":"2024-05-03T17:20:33.284178+0000","last_became_peered":"2024-05-03T17:20:33.284178+0000","last_unstale":"2024-05-03T17:21:07.594059+0000","last_undegraded":"2024-05-03T17:21:07.594059+0000","last_fullsized":"2024-05-03T17:21:07.594059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:45:57.315989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597013+0000","last_change":"2024-05-03T17:20:37.201286+0000","last_active":"2024-05-03T17:21:07.597013+0000","last_peered":"2024-05-03T17:21:07.597013+0000","last_clean":"2024-05-03T17:21:07.597013+0000","last_became_active":"2024-05-03T17:20:37.200879+0000","last_became_peered":"2024-05-03T17:20:37.200879+0000","last_unstale":"2024-05-03T17:21:07.597013+0000","last_undegraded":"2024-05-03T17:21:07.597013+0000","last_fullsized":"2024-05-03T17:21:07.597013+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:25:49.936431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604390+0000","last_change":"2024-05-03T17:20:33.165850+0000","last_active":"2024-05-03T17:21:07.604390+0000","last_peered":"2024-05-03T17:21:07.604390+0000","last_clean":"2024-05-03T17:21:07.604390+0000","last_became_active":"2024-05-03T17:20:33.165478+0000","last_became_peered":"2024-05-03T17:20:33.165478+0000","last_unstale":"2024-05-03T17:21:07.604390+0000","last_undegraded":"2024-05-03T17:21:07.604390+0000","last_fullsized":"2024-05-03T17:21:07.604390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:22:36.224538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596692+0000","last_change":"2024-05-03T17:20:37.210896+0000","last_active":"2024-05-03T17:21:07.596692+0000","last_peered":"2024-05-03T17:21:07.596692+0000","last_clean":"2024-05-03T17:21:07.596692+0000","last_became_active":"2024-05-03T17:20:37.209150+0000","last_became_peered":"2024-05-03T17:20:37.209150+0000","last_unstale":"2024-05-03T17:21:07.596692+0000","last_undegraded":"2024-05-03T17:21:07.596692+0000","last_fullsized":"2024-05-03T17:21:07.596692+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:17:01.870395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594502+0000","last_change":"2024-05-03T17:20:33.169631+0000","last_active":"2024-05-03T17:21:07.594502+0000","last_peered":"2024-05-03T17:21:07.594502+0000","last_clean":"2024-05-03T17:21:07.594502+0000","last_became_active":"2024-05-03T17:20:33.169293+0000","last_became_peered":"2024-05-03T17:20:33.169293+0000","last_unstale":"2024-05-03T17:21:07.594502+0000","last_undegraded":"2024-05-03T17:21:07.594502+0000","last_fullsized":"2024-05-03T17:21:07.594502+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:20:29.988150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597243+0000","last_change":"2024-05-03T17:20:37.201347+0000","last_active":"2024-05-03T17:21:07.597243+0000","last_peered":"2024-05-03T17:21:07.597243+0000","last_clean":"2024-05-03T17:21:07.597243+0000","last_became_active":"2024-05-03T17:20:37.200999+0000","last_became_peered":"2024-05-03T17:20:37.200999+0000","last_unstale":"2024-05-03T17:21:07.597243+0000","last_undegraded":"2024-05-03T17:21:07.597243+0000","last_fullsized":"2024-05-03T17:21:07.597243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:34:23.027957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595391+0000","last_change":"2024-05-03T17:20:37.206379+0000","last_active":"2024-05-03T17:21:07.595391+0000","last_peered":"2024-05-03T17:21:07.595391+0000","last_clean":"2024-05-03T17:21:07.595391+0000","last_became_active":"2024-05-03T17:20:37.206048+0000","last_became_peered":"2024-05-03T17:20:37.206048+0000","last_unstale":"2024-05-03T17:21:07.595391+0000","last_undegraded":"2024-05-03T17:21:07.595391+0000","last_fullsized":"2024-05-03T17:21:07.595391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:46:02.486170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593822+0000","last_change":"2024-05-03T17:20:37.205997+0000","last_active":"2024-05-03T17:21:07.593822+0000","last_peered":"2024-05-03T17:21:07.593822+0000","last_clean":"2024-05-03T17:21:07.593822+0000","last_became_active":"2024-05-03T17:20:37.205602+0000","last_became_peered":"2024-05-03T17:20:37.205602+0000","last_unstale":"2024-05-03T17:21:07.593822+0000","last_undegraded":"2024-05-03T17:21:07.593822+0000","last_fullsized":"2024-05-03T17:21:07.593822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:16:04.929294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593673+0000","last_change":"2024-05-03T17:20:37.199994+0000","last_active":"2024-05-03T17:21:07.593673+0000","last_peered":"2024-05-03T17:21:07.593673+0000","last_clean":"2024-05-03T17:21:07.593673+0000","last_became_active":"2024-05-03T17:20:37.199653+0000","last_became_peered":"2024-05-03T17:20:37.199653+0000","last_unstale":"2024-05-03T17:21:07.593673+0000","last_undegraded":"2024-05-03T17:21:07.593673+0000","last_fullsized":"2024-05-03T17:21:07.593673+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:23:24.945036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598001+0000","last_change":"2024-05-03T17:20:37.198701+0000","last_active":"2024-05-03T17:21:07.598001+0000","last_peered":"2024-05-03T17:21:07.598001+0000","last_clean":"2024-05-03T17:21:07.598001+0000","last_became_active":"2024-05-03T17:20:37.198374+0000","last_became_peered":"2024-05-03T17:20:37.198374+0000","last_unstale":"2024-05-03T17:21:07.598001+0000","last_undegraded":"2024-05-03T17:21:07.598001+0000","last_fullsized":"2024-05-03T17:21:07.598001+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:33:26.744746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599746+0000","last_change":"2024-05-03T17:21:07.599746+0000","last_active":"2024-05-03T17:21:07.599746+0000","last_peered":"2024-05-03T17:21:07.599746+0000","last_clean":"2024-05-03T17:21:07.599746+0000","last_became_active":"2024-05-03T17:21:07.598862+0000","last_became_peered":"2024-05-03T17:21:07.598862+0000","last_unstale":"2024-05-03T17:21:07.599746+0000","last_undegraded":"2024-05-03T17:21:07.599746+0000","last_fullsized":"2024-05-03T17:21:07.599746+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:46:31.214548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596253+0000","last_change":"2024-05-03T17:20:37.207194+0000","last_active":"2024-05-03T17:21:07.596253+0000","last_peered":"2024-05-03T17:21:07.596253+0000","last_clean":"2024-05-03T17:21:07.596253+0000","last_became_active":"2024-05-03T17:20:37.206833+0000","last_became_peered":"2024-05-03T17:20:37.206833+0000","last_unstale":"2024-05-03T17:21:07.596253+0000","last_undegraded":"2024-05-03T17:21:07.596253+0000","last_fullsized":"2024-05-03T17:21:07.596253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:29:32.212841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594069+0000","last_change":"2024-05-03T17:20:37.203613+0000","last_active":"2024-05-03T17:21:07.594069+0000","last_peered":"2024-05-03T17:21:07.594069+0000","last_clean":"2024-05-03T17:21:07.594069+0000","last_became_active":"2024-05-03T17:20:37.203265+0000","last_became_peered":"2024-05-03T17:20:37.203265+0000","last_unstale":"2024-05-03T17:21:07.594069+0000","last_undegraded":"2024-05-03T17:21:07.594069+0000","last_fullsized":"2024-05-03T17:21:07.594069+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:12:35.816804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592409+0000","last_change":"2024-05-03T17:20:37.206208+0000","last_active":"2024-05-03T17:21:07.592409+0000","last_peered":"2024-05-03T17:21:07.592409+0000","last_clean":"2024-05-03T17:21:07.592409+0000","last_became_active":"2024-05-03T17:20:37.205859+0000","last_became_peered":"2024-05-03T17:20:37.205859+0000","last_unstale":"2024-05-03T17:21:07.592409+0000","last_undegraded":"2024-05-03T17:21:07.592409+0000","last_fullsized":"2024-05-03T17:21:07.592409+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:57:55.311154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604102+0000","last_change":"2024-05-03T17:20:37.197239+0000","last_active":"2024-05-03T17:21:07.604102+0000","last_peered":"2024-05-03T17:21:07.604102+0000","last_clean":"2024-05-03T17:21:07.604102+0000","last_became_active":"2024-05-03T17:20:37.196903+0000","last_became_peered":"2024-05-03T17:20:37.196903+0000","last_unstale":"2024-05-03T17:21:07.604102+0000","last_undegraded":"2024-05-03T17:21:07.604102+0000","last_fullsized":"2024-05-03T17:21:07.604102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:29:53.020573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592086+0000","last_change":"2024-05-03T17:20:37.211295+0000","last_active":"2024-05-03T17:21:07.592086+0000","last_peered":"2024-05-03T17:21:07.592086+0000","last_clean":"2024-05-03T17:21:07.592086+0000","last_became_active":"2024-05-03T17:20:37.210939+0000","last_became_peered":"2024-05-03T17:20:37.210939+0000","last_unstale":"2024-05-03T17:21:07.592086+0000","last_undegraded":"2024-05-03T17:21:07.592086+0000","last_fullsized":"2024-05-03T17:21:07.592086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:45:58.770190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598703+0000","last_change":"2024-05-03T17:20:37.209581+0000","last_active":"2024-05-03T17:21:07.598703+0000","last_peered":"2024-05-03T17:21:07.598703+0000","last_clean":"2024-05-03T17:21:07.598703+0000","last_became_active":"2024-05-03T17:20:37.209126+0000","last_became_peered":"2024-05-03T17:20:37.209126+0000","last_unstale":"2024-05-03T17:21:07.598703+0000","last_undegraded":"2024-05-03T17:21:07.598703+0000","last_fullsized":"2024-05-03T17:21:07.598703+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:47:15.485725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595388+0000","last_change":"2024-05-03T17:20:37.204258+0000","last_active":"2024-05-03T17:21:07.595388+0000","last_peered":"2024-05-03T17:21:07.595388+0000","last_clean":"2024-05-03T17:21:07.595388+0000","last_became_active":"2024-05-03T17:20:37.202405+0000","last_became_peered":"2024-05-03T17:20:37.202405+0000","last_unstale":"2024-05-03T17:21:07.595388+0000","last_undegraded":"2024-05-03T17:21:07.595388+0000","last_fullsized":"2024-05-03T17:21:07.595388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:36:35.946607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595536+0000","last_change":"2024-05-03T17:20:37.208909+0000","last_active":"2024-05-03T17:21:07.595536+0000","last_peered":"2024-05-03T17:21:07.595536+0000","last_clean":"2024-05-03T17:21:07.595536+0000","last_became_active":"2024-05-03T17:20:37.208540+0000","last_became_peered":"2024-05-03T17:20:37.208540+0000","last_unstale":"2024-05-03T17:21:07.595536+0000","last_undegraded":"2024-05-03T17:21:07.595536+0000","last_fullsized":"2024-05-03T17:21:07.595536+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:32:41.700487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":213988,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":55834,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":626,"num_read_kb":369,"num_write":162,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":167502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678058,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":876544,"data_stored":428314,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874286,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":1071606,"data_compressed":42001,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037812,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":428528,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234042,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":428426,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430270,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":1020378,"data_compressed":42001,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593796,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32040,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713112,"statfs":{"total":95995035648,"available":95962226688,"internally_reserved":0,"allocated":876544,"data_stored":427917,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":479543,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738466,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":897024,"data_stored":429188,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":481284,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":892928,"data_stored":428228,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":880640,"data_stored":428354,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790026,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":1020062,"data_compressed":42001,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34582,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34582,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":51833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":53574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":53411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-03T17:22:43.438 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph pg dump --format=json 2024-05-03T17:22:43.692 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:43.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:43 smithi067 ceph-mon[29731]: pgmap v359: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:22:43.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:43.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:43 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:43.703 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:43 smithi067 ceph-mon[29731]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:22:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:43 smithi070 ceph-mon[33250]: pgmap v359: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:22:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:43 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:43.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:43 smithi070 ceph-mon[33250]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:22:44.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:43 smithi082 ceph-mon[33535]: pgmap v359: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:22:44.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:44.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:43 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:44.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:43 smithi082 ceph-mon[33535]: from='client.14883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:22:44.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:22:44.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:22:44.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:44.835 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:22:44.835 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:22:44.835 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:45.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:22:45.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:22:45.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:22:45.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.483+0000 7f8abde7a700 1 -- 172.21.15.67:0/416425536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab8109590 msgr2=0x7f8ab8109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:45.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.483+0000 7f8abde7a700 1 --2- 172.21.15.67:0/416425536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab8109590 0x7f8ab8109970 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8aa80099c0 tx=0x7f8aa8009cd0 comp rx=0 tx=0).stop 2024-05-03T17:22:45.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.483+0000 7f8abde7a700 1 -- 172.21.15.67:0/416425536 shutdown_connections 2024-05-03T17:22:45.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.483+0000 7f8abde7a700 1 --2- 172.21.15.67:0/416425536 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ab806c630 0x7f8ab8110360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:45.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.483+0000 7f8abde7a700 1 --2- 172.21.15.67:0/416425536 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab8109590 0x7f8ab8109970 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:45.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.483+0000 7f8abde7a700 1 --2- 172.21.15.67:0/416425536 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ab806bc70 0x7f8ab806c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.483+0000 7f8abde7a700 1 -- 172.21.15.67:0/416425536 >> 172.21.15.67:0/416425536 conn(0x7f8ab8079440 msgr2=0x7f8ab8079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.483+0000 7f8abde7a700 1 -- 172.21.15.67:0/416425536 shutdown_connections 2024-05-03T17:22:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.483+0000 7f8abde7a700 1 -- 172.21.15.67:0/416425536 wait complete. 2024-05-03T17:22:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.483+0000 7f8abde7a700 1 Processor -- start 2024-05-03T17:22:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.483+0000 7f8abde7a700 1 -- start start 2024-05-03T17:22:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8abde7a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab806bc70 0x7f8ab819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8abde7a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ab806c630 0x7f8ab819fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8abde7a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ab8109590 0x7f8ab8199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:45.495 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8abde7a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab8113680 con 0x7f8ab806bc70 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8abde7a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab8113500 con 0x7f8ab8109590 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8abde7a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8ab8113800 con 0x7f8ab806c630 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab806bc70 0x7f8ab819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab77fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab806bc70 0x7f8ab819f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:59068/0 (socket says 172.21.15.67:59068) 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab77fe700 1 -- 172.21.15.67:0/1644730838 learned_addr learned my addr 172.21.15.67:0/1644730838 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab6ffd700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ab806c630 0x7f8ab819fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab7fff700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ab8109590 0x7f8ab8199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab77fe700 1 -- 172.21.15.67:0/1644730838 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ab806c630 msgr2=0x7f8ab819fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab77fe700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ab806c630 0x7f8ab819fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab77fe700 1 -- 172.21.15.67:0/1644730838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ab8109590 msgr2=0x7f8ab8199960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab77fe700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ab8109590 0x7f8ab8199960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab77fe700 1 -- 172.21.15.67:0/1644730838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ab819a0c0 con 0x7f8ab806bc70 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab6ffd700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ab806c630 0x7f8ab819fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab77fe700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab806bc70 0x7f8ab819f7a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800b580 tx=0x7f8aa80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab4ff9700 1 -- 172.21.15.67:0/1644730838 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa8027070 con 0x7f8ab806bc70 2024-05-03T17:22:45.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab4ff9700 1 -- 172.21.15.67:0/1644730838 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8aa8004020 con 0x7f8ab806bc70 2024-05-03T17:22:45.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.484+0000 7f8ab4ff9700 1 -- 172.21.15.67:0/1644730838 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa8022e30 con 0x7f8ab806bc70 2024-05-03T17:22:45.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.485+0000 7f8abde7a700 1 -- 172.21.15.67:0/1644730838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ab819a350 con 0x7f8ab806bc70 2024-05-03T17:22:45.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.485+0000 7f8abde7a700 1 -- 172.21.15.67:0/1644730838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ab81a6670 con 0x7f8ab806bc70 2024-05-03T17:22:45.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.486+0000 7f8abde7a700 1 -- 172.21.15.67:0/1644730838 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ab810ba20 con 0x7f8ab806bc70 2024-05-03T17:22:45.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.487+0000 7f8ab4ff9700 1 -- 172.21.15.67:0/1644730838 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f8aa8022590 con 0x7f8ab806bc70 2024-05-03T17:22:45.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.487+0000 7f8ab4ff9700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8aa00b2990 0x7f8aa00b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:45.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.488+0000 7f8ab6ffd700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8aa00b2990 0x7f8aa00b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:45.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.488+0000 7f8ab4ff9700 1 -- 172.21.15.67:0/1644730838 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f8aa80e2fe0 con 0x7f8ab806bc70 2024-05-03T17:22:45.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.488+0000 7f8ab6ffd700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8aa00b2990 0x7f8aa00b4e50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8aac005950 tx=0x7f8aac0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:45.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.492+0000 7f8ab4ff9700 1 -- 172.21.15.67:0/1644730838 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f8aa8013070 con 0x7f8ab806bc70 2024-05-03T17:22:45.700 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.700+0000 7f8abde7a700 1 -- 172.21.15.67:0/1644730838 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8ab819ae90 con 0x7f8aa00b2990 2024-05-03T17:22:45.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.707+0000 7f8ab4ff9700 1 -- 172.21.15.67:0/1644730838 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258922 (secure 0 0 0) 0x7f8ab819ae90 con 0x7f8aa00b2990 2024-05-03T17:22:45.711 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:22:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.710+0000 7f8abde7a700 1 -- 172.21.15.67:0/1644730838 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8aa00b2990 msgr2=0x7f8aa00b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.710+0000 7f8abde7a700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8aa00b2990 0x7f8aa00b4e50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f8aac005950 tx=0x7f8aac0058e0 comp rx=0 tx=0).stop 2024-05-03T17:22:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.710+0000 7f8abde7a700 1 -- 172.21.15.67:0/1644730838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab806bc70 msgr2=0x7f8ab819f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:45.713 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.710+0000 7f8abde7a700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab806bc70 0x7f8ab819f7a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8aa800b580 tx=0x7f8aa80047b0 comp rx=0 tx=0).stop 2024-05-03T17:22:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.710+0000 7f8abde7a700 1 -- 172.21.15.67:0/1644730838 shutdown_connections 2024-05-03T17:22:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.710+0000 7f8abde7a700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8aa00b2990 0x7f8aa00b4e50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.710+0000 7f8abde7a700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8ab8109590 0x7f8ab8199960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.710+0000 7f8abde7a700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8ab806bc70 0x7f8ab819f7a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.710+0000 7f8abde7a700 1 --2- 172.21.15.67:0/1644730838 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8ab806c630 0x7f8ab819fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.710+0000 7f8abde7a700 1 -- 172.21.15.67:0/1644730838 >> 172.21.15.67:0/1644730838 conn(0x7f8ab8079440 msgr2=0x7f8ab8100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.710+0000 7f8abde7a700 1 -- 172.21.15.67:0/1644730838 shutdown_connections 2024-05-03T17:22:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:45.710+0000 7f8abde7a700 1 -- 172.21.15.67:0/1644730838 wait complete. 2024-05-03T17:22:45.714 INFO:teuthology.orchestra.run.smithi067.stderr:dumped all 2024-05-03T17:22:45.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:45 smithi067 ceph-mon[29731]: pgmap v360: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-03T17:22:46.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:45 smithi082 ceph-mon[33535]: pgmap v360: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-03T17:22:46.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:45 smithi070 ceph-mon[33250]: pgmap v360: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-03T17:22:46.404 INFO:teuthology.orchestra.run.smithi067.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-05-03T17:22:44.448283+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":646436,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":727,"num_read_kb":448,"num_write":287,"num_write_kb":2254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":2951862,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":231,"ondisk_log_size":231,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":359904,"kb_used_data":10728,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581920,"statfs":{"total":1151940427776,"available":1151571886080,"internally_reserved":0,"allocated":10985472,"data_stored":7071828,"data_compressed":178158,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19143,"internal_metadata":357414201},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":85,"num_read_kb":45,"num_write":3,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002801"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595780+0000","last_change":"2024-05-03T17:20:37.204469+0000","last_active":"2024-05-03T17:21:07.595780+0000","last_peered":"2024-05-03T17:21:07.595780+0000","last_clean":"2024-05-03T17:21:07.595780+0000","last_became_active":"2024-05-03T17:20:37.204098+0000","last_became_peered":"2024-05-03T17:20:37.204098+0000","last_unstale":"2024-05-03T17:21:07.595780+0000","last_undegraded":"2024-05-03T17:21:07.595780+0000","last_fullsized":"2024-05-03T17:21:07.595780+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:03:29.875319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599143+0000","last_change":"2024-05-03T17:20:37.201171+0000","last_active":"2024-05-03T17:21:07.599143+0000","last_peered":"2024-05-03T17:21:07.599143+0000","last_clean":"2024-05-03T17:21:07.599143+0000","last_became_active":"2024-05-03T17:20:37.200852+0000","last_became_peered":"2024-05-03T17:20:37.200852+0000","last_unstale":"2024-05-03T17:21:07.599143+0000","last_undegraded":"2024-05-03T17:21:07.599143+0000","last_fullsized":"2024-05-03T17:21:07.599143+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:12:06.549919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598965+0000","last_change":"2024-05-03T17:20:37.201925+0000","last_active":"2024-05-03T17:21:07.598965+0000","last_peered":"2024-05-03T17:21:07.598965+0000","last_clean":"2024-05-03T17:21:07.598965+0000","last_became_active":"2024-05-03T17:20:37.201619+0000","last_became_peered":"2024-05-03T17:20:37.201619+0000","last_unstale":"2024-05-03T17:21:07.598965+0000","last_undegraded":"2024-05-03T17:21:07.598965+0000","last_fullsized":"2024-05-03T17:21:07.598965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:50:19.019887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603188+0000","last_change":"2024-05-03T17:20:37.203535+0000","last_active":"2024-05-03T17:21:07.603188+0000","last_peered":"2024-05-03T17:21:07.603188+0000","last_clean":"2024-05-03T17:21:07.603188+0000","last_became_active":"2024-05-03T17:20:37.203175+0000","last_became_peered":"2024-05-03T17:20:37.203175+0000","last_unstale":"2024-05-03T17:21:07.603188+0000","last_undegraded":"2024-05-03T17:21:07.603188+0000","last_fullsized":"2024-05-03T17:21:07.603188+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:59:41.026709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597618+0000","last_change":"2024-05-03T17:20:37.200283+0000","last_active":"2024-05-03T17:21:07.597618+0000","last_peered":"2024-05-03T17:21:07.597618+0000","last_clean":"2024-05-03T17:21:07.597618+0000","last_became_active":"2024-05-03T17:20:37.199105+0000","last_became_peered":"2024-05-03T17:20:37.199105+0000","last_unstale":"2024-05-03T17:21:07.597618+0000","last_undegraded":"2024-05-03T17:21:07.597618+0000","last_fullsized":"2024-05-03T17:21:07.597618+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:56:39.990497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597382+0000","last_change":"2024-05-03T17:21:07.597382+0000","last_active":"2024-05-03T17:21:07.597382+0000","last_peered":"2024-05-03T17:21:07.597382+0000","last_clean":"2024-05-03T17:21:07.597382+0000","last_became_active":"2024-05-03T17:21:07.596935+0000","last_became_peered":"2024-05-03T17:21:07.596935+0000","last_unstale":"2024-05-03T17:21:07.597382+0000","last_undegraded":"2024-05-03T17:21:07.597382+0000","last_fullsized":"2024-05-03T17:21:07.597382+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T19:30:50.200783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,6],"acting":[10,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596543+0000","last_change":"2024-05-03T17:20:37.211378+0000","last_active":"2024-05-03T17:21:07.596543+0000","last_peered":"2024-05-03T17:21:07.596543+0000","last_clean":"2024-05-03T17:21:07.596543+0000","last_became_active":"2024-05-03T17:20:37.210684+0000","last_became_peered":"2024-05-03T17:20:37.210684+0000","last_unstale":"2024-05-03T17:21:07.596543+0000","last_undegraded":"2024-05-03T17:21:07.596543+0000","last_fullsized":"2024-05-03T17:21:07.596543+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:35:24.737186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592032+0000","last_change":"2024-05-03T17:20:37.202989+0000","last_active":"2024-05-03T17:21:07.592032+0000","last_peered":"2024-05-03T17:21:07.592032+0000","last_clean":"2024-05-03T17:21:07.592032+0000","last_became_active":"2024-05-03T17:20:37.199975+0000","last_became_peered":"2024-05-03T17:20:37.199975+0000","last_unstale":"2024-05-03T17:21:07.592032+0000","last_undegraded":"2024-05-03T17:21:07.592032+0000","last_fullsized":"2024-05-03T17:21:07.592032+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:13:43.875472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598772+0000","last_change":"2024-05-03T17:20:37.206754+0000","last_active":"2024-05-03T17:21:07.598772+0000","last_peered":"2024-05-03T17:21:07.598772+0000","last_clean":"2024-05-03T17:21:07.598772+0000","last_became_active":"2024-05-03T17:20:37.206423+0000","last_became_peered":"2024-05-03T17:20:37.206423+0000","last_unstale":"2024-05-03T17:21:07.598772+0000","last_undegraded":"2024-05-03T17:21:07.598772+0000","last_fullsized":"2024-05-03T17:21:07.598772+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:10:30.227969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593231+0000","last_change":"2024-05-03T17:20:37.211868+0000","last_active":"2024-05-03T17:21:07.593231+0000","last_peered":"2024-05-03T17:21:07.593231+0000","last_clean":"2024-05-03T17:21:07.593231+0000","last_became_active":"2024-05-03T17:20:37.211486+0000","last_became_peered":"2024-05-03T17:20:37.211486+0000","last_unstale":"2024-05-03T17:21:07.593231+0000","last_undegraded":"2024-05-03T17:21:07.593231+0000","last_fullsized":"2024-05-03T17:21:07.593231+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:41:03.966446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596847+0000","last_change":"2024-05-03T17:20:37.206511+0000","last_active":"2024-05-03T17:21:07.596847+0000","last_peered":"2024-05-03T17:21:07.596847+0000","last_clean":"2024-05-03T17:21:07.596847+0000","last_became_active":"2024-05-03T17:20:37.206180+0000","last_became_peered":"2024-05-03T17:20:37.206180+0000","last_unstale":"2024-05-03T17:21:07.596847+0000","last_undegraded":"2024-05-03T17:21:07.596847+0000","last_fullsized":"2024-05-03T17:21:07.596847+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:59:07.277623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603068+0000","last_change":"2024-05-03T17:20:37.204842+0000","last_active":"2024-05-03T17:21:07.603068+0000","last_peered":"2024-05-03T17:21:07.603068+0000","last_clean":"2024-05-03T17:21:07.603068+0000","last_became_active":"2024-05-03T17:20:37.204275+0000","last_became_peered":"2024-05-03T17:20:37.204275+0000","last_unstale":"2024-05-03T17:21:07.603068+0000","last_undegraded":"2024-05-03T17:21:07.603068+0000","last_fullsized":"2024-05-03T17:21:07.603068+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:27:00.021413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597215+0000","last_change":"2024-05-03T17:21:07.597215+0000","last_active":"2024-05-03T17:21:07.597215+0000","last_peered":"2024-05-03T17:21:07.597215+0000","last_clean":"2024-05-03T17:21:07.597215+0000","last_became_active":"2024-05-03T17:21:07.596833+0000","last_became_peered":"2024-05-03T17:21:07.596833+0000","last_unstale":"2024-05-03T17:21:07.597215+0000","last_undegraded":"2024-05-03T17:21:07.597215+0000","last_fullsized":"2024-05-03T17:21:07.597215+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:09:41.399761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,9],"acting":[6,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597704+0000","last_change":"2024-05-03T17:20:37.200759+0000","last_active":"2024-05-03T17:21:07.597704+0000","last_peered":"2024-05-03T17:21:07.597704+0000","last_clean":"2024-05-03T17:21:07.597704+0000","last_became_active":"2024-05-03T17:20:37.200410+0000","last_became_peered":"2024-05-03T17:20:37.200410+0000","last_unstale":"2024-05-03T17:21:07.597704+0000","last_undegraded":"2024-05-03T17:21:07.597704+0000","last_fullsized":"2024-05-03T17:21:07.597704+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:51:03.719792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597446+0000","last_change":"2024-05-03T17:20:37.204534+0000","last_active":"2024-05-03T17:21:07.597446+0000","last_peered":"2024-05-03T17:21:07.597446+0000","last_clean":"2024-05-03T17:21:07.597446+0000","last_became_active":"2024-05-03T17:20:37.204167+0000","last_became_peered":"2024-05-03T17:20:37.204167+0000","last_unstale":"2024-05-03T17:21:07.597446+0000","last_undegraded":"2024-05-03T17:21:07.597446+0000","last_fullsized":"2024-05-03T17:21:07.597446+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:31:35.730465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598463+0000","last_change":"2024-05-03T17:20:37.207658+0000","last_active":"2024-05-03T17:21:07.598463+0000","last_peered":"2024-05-03T17:21:07.598463+0000","last_clean":"2024-05-03T17:21:07.598463+0000","last_became_active":"2024-05-03T17:20:37.207313+0000","last_became_peered":"2024-05-03T17:20:37.207313+0000","last_unstale":"2024-05-03T17:21:07.598463+0000","last_undegraded":"2024-05-03T17:21:07.598463+0000","last_fullsized":"2024-05-03T17:21:07.598463+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:47:21.287559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595876+0000","last_change":"2024-05-03T17:20:37.205021+0000","last_active":"2024-05-03T17:21:07.595876+0000","last_peered":"2024-05-03T17:21:07.595876+0000","last_clean":"2024-05-03T17:21:07.595876+0000","last_became_active":"2024-05-03T17:20:37.204630+0000","last_became_peered":"2024-05-03T17:20:37.204630+0000","last_unstale":"2024-05-03T17:21:07.595876+0000","last_undegraded":"2024-05-03T17:21:07.595876+0000","last_fullsized":"2024-05-03T17:21:07.595876+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:03:00.318648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593860+0000","last_change":"2024-05-03T17:20:37.201829+0000","last_active":"2024-05-03T17:21:07.593860+0000","last_peered":"2024-05-03T17:21:07.593860+0000","last_clean":"2024-05-03T17:21:07.593860+0000","last_became_active":"2024-05-03T17:20:37.201465+0000","last_became_peered":"2024-05-03T17:20:37.201465+0000","last_unstale":"2024-05-03T17:21:07.593860+0000","last_undegraded":"2024-05-03T17:21:07.593860+0000","last_fullsized":"2024-05-03T17:21:07.593860+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:05:46.844150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594023+0000","last_change":"2024-05-03T17:20:37.199138+0000","last_active":"2024-05-03T17:21:07.594023+0000","last_peered":"2024-05-03T17:21:07.594023+0000","last_clean":"2024-05-03T17:21:07.594023+0000","last_became_active":"2024-05-03T17:20:37.198722+0000","last_became_peered":"2024-05-03T17:20:37.198722+0000","last_unstale":"2024-05-03T17:21:07.594023+0000","last_undegraded":"2024-05-03T17:21:07.594023+0000","last_fullsized":"2024-05-03T17:21:07.594023+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:12:04.764356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'9","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594031+0000","last_change":"2024-05-03T17:20:33.168111+0000","last_active":"2024-05-03T17:21:07.594031+0000","last_peered":"2024-05-03T17:21:07.594031+0000","last_clean":"2024-05-03T17:21:07.594031+0000","last_became_active":"2024-05-03T17:20:33.167892+0000","last_became_peered":"2024-05-03T17:20:33.167892+0000","last_unstale":"2024-05-03T17:21:07.594031+0000","last_undegraded":"2024-05-03T17:21:07.594031+0000","last_fullsized":"2024-05-03T17:21:07.594031+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:55:45.235287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592267+0000","last_change":"2024-05-03T17:20:37.208673+0000","last_active":"2024-05-03T17:21:07.592267+0000","last_peered":"2024-05-03T17:21:07.592267+0000","last_clean":"2024-05-03T17:21:07.592267+0000","last_became_active":"2024-05-03T17:20:37.208360+0000","last_became_peered":"2024-05-03T17:20:37.208360+0000","last_unstale":"2024-05-03T17:21:07.592267+0000","last_undegraded":"2024-05-03T17:21:07.592267+0000","last_fullsized":"2024-05-03T17:21:07.592267+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:42:02.966225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592646+0000","last_change":"2024-05-03T17:20:33.284052+0000","last_active":"2024-05-03T17:21:07.592646+0000","last_peered":"2024-05-03T17:21:07.592646+0000","last_clean":"2024-05-03T17:21:07.592646+0000","last_became_active":"2024-05-03T17:20:33.283635+0000","last_became_peered":"2024-05-03T17:20:33.283635+0000","last_unstale":"2024-05-03T17:21:07.592646+0000","last_undegraded":"2024-05-03T17:21:07.592646+0000","last_fullsized":"2024-05-03T17:21:07.592646+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:17:51.219359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598851+0000","last_change":"2024-05-03T17:20:37.197321+0000","last_active":"2024-05-03T17:21:07.598851+0000","last_peered":"2024-05-03T17:21:07.598851+0000","last_clean":"2024-05-03T17:21:07.598851+0000","last_became_active":"2024-05-03T17:20:37.196569+0000","last_became_peered":"2024-05-03T17:20:37.196569+0000","last_unstale":"2024-05-03T17:21:07.598851+0000","last_undegraded":"2024-05-03T17:21:07.598851+0000","last_fullsized":"2024-05-03T17:21:07.598851+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:38:03.274058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598373+0000","last_change":"2024-05-03T17:20:33.168828+0000","last_active":"2024-05-03T17:21:07.598373+0000","last_peered":"2024-05-03T17:21:07.598373+0000","last_clean":"2024-05-03T17:21:07.598373+0000","last_became_active":"2024-05-03T17:20:33.168408+0000","last_became_peered":"2024-05-03T17:20:33.168408+0000","last_unstale":"2024-05-03T17:21:07.598373+0000","last_undegraded":"2024-05-03T17:21:07.598373+0000","last_fullsized":"2024-05-03T17:21:07.598373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:50:57.790353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598587+0000","last_change":"2024-05-03T17:20:37.199853+0000","last_active":"2024-05-03T17:21:07.598587+0000","last_peered":"2024-05-03T17:21:07.598587+0000","last_clean":"2024-05-03T17:21:07.598587+0000","last_became_active":"2024-05-03T17:20:37.199526+0000","last_became_peered":"2024-05-03T17:20:37.199526+0000","last_unstale":"2024-05-03T17:21:07.598587+0000","last_undegraded":"2024-05-03T17:21:07.598587+0000","last_fullsized":"2024-05-03T17:21:07.598587+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:42:45.932795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597857+0000","last_change":"2024-05-03T17:20:33.168050+0000","last_active":"2024-05-03T17:21:07.597857+0000","last_peered":"2024-05-03T17:21:07.597857+0000","last_clean":"2024-05-03T17:21:07.597857+0000","last_became_active":"2024-05-03T17:20:33.167479+0000","last_became_peered":"2024-05-03T17:20:33.167479+0000","last_unstale":"2024-05-03T17:21:07.597857+0000","last_undegraded":"2024-05-03T17:21:07.597857+0000","last_fullsized":"2024-05-03T17:21:07.597857+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:39:32.626940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604399+0000","last_change":"2024-05-03T17:21:07.603996+0000","last_active":"2024-05-03T17:21:07.604399+0000","last_peered":"2024-05-03T17:21:07.604399+0000","last_clean":"2024-05-03T17:21:07.604399+0000","last_became_active":"2024-05-03T17:21:07.603647+0000","last_became_peered":"2024-05-03T17:21:07.603647+0000","last_unstale":"2024-05-03T17:21:07.604399+0000","last_undegraded":"2024-05-03T17:21:07.604399+0000","last_fullsized":"2024-05-03T17:21:07.604399+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:31:07.956642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1b","version":"79'6","reported_seq":139,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:22:40.731321+0000","last_change":"2024-05-03T17:20:33.166560+0000","last_active":"2024-05-03T17:22:40.731321+0000","last_peered":"2024-05-03T17:22:40.731321+0000","last_clean":"2024-05-03T17:22:40.731321+0000","last_became_active":"2024-05-03T17:20:33.166136+0000","last_became_peered":"2024-05-03T17:20:33.166136+0000","last_unstale":"2024-05-03T17:22:40.731321+0000","last_undegraded":"2024-05-03T17:22:40.731321+0000","last_fullsized":"2024-05-03T17:22:40.731321+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:38:54.132364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":107,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597525+0000","last_change":"2024-05-03T17:20:37.204336+0000","last_active":"2024-05-03T17:21:07.597525+0000","last_peered":"2024-05-03T17:21:07.597525+0000","last_clean":"2024-05-03T17:21:07.597525+0000","last_became_active":"2024-05-03T17:20:37.203962+0000","last_became_peered":"2024-05-03T17:20:37.203962+0000","last_unstale":"2024-05-03T17:21:07.597525+0000","last_undegraded":"2024-05-03T17:21:07.597525+0000","last_fullsized":"2024-05-03T17:21:07.597525+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:29:48.081362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592586+0000","last_change":"2024-05-03T17:20:33.171046+0000","last_active":"2024-05-03T17:21:07.592586+0000","last_peered":"2024-05-03T17:21:07.592586+0000","last_clean":"2024-05-03T17:21:07.592586+0000","last_became_active":"2024-05-03T17:20:33.169410+0000","last_became_peered":"2024-05-03T17:20:33.169410+0000","last_unstale":"2024-05-03T17:21:07.592586+0000","last_undegraded":"2024-05-03T17:21:07.592586+0000","last_fullsized":"2024-05-03T17:21:07.592586+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:45:05.987672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604346+0000","last_change":"2024-05-03T17:20:37.210695+0000","last_active":"2024-05-03T17:21:07.604346+0000","last_peered":"2024-05-03T17:21:07.604346+0000","last_clean":"2024-05-03T17:21:07.604346+0000","last_became_active":"2024-05-03T17:20:37.210285+0000","last_became_peered":"2024-05-03T17:20:37.210285+0000","last_unstale":"2024-05-03T17:21:07.604346+0000","last_undegraded":"2024-05-03T17:21:07.604346+0000","last_fullsized":"2024-05-03T17:21:07.604346+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:10:52.013017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595054+0000","last_change":"2024-05-03T17:20:33.163997+0000","last_active":"2024-05-03T17:21:07.595054+0000","last_peered":"2024-05-03T17:21:07.595054+0000","last_clean":"2024-05-03T17:21:07.595054+0000","last_became_active":"2024-05-03T17:20:33.163653+0000","last_became_peered":"2024-05-03T17:20:33.163653+0000","last_unstale":"2024-05-03T17:21:07.595054+0000","last_undegraded":"2024-05-03T17:21:07.595054+0000","last_fullsized":"2024-05-03T17:21:07.595054+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:55:51.762133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592099+0000","last_change":"2024-05-03T17:20:37.195967+0000","last_active":"2024-05-03T17:21:07.592099+0000","last_peered":"2024-05-03T17:21:07.592099+0000","last_clean":"2024-05-03T17:21:07.592099+0000","last_became_active":"2024-05-03T17:20:37.195652+0000","last_became_peered":"2024-05-03T17:20:37.195652+0000","last_unstale":"2024-05-03T17:21:07.592099+0000","last_undegraded":"2024-05-03T17:21:07.592099+0000","last_fullsized":"2024-05-03T17:21:07.592099+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:34:37.381525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603936+0000","last_change":"2024-05-03T17:20:33.167378+0000","last_active":"2024-05-03T17:21:07.603936+0000","last_peered":"2024-05-03T17:21:07.603936+0000","last_clean":"2024-05-03T17:21:07.603936+0000","last_became_active":"2024-05-03T17:20:33.167000+0000","last_became_peered":"2024-05-03T17:20:33.167000+0000","last_unstale":"2024-05-03T17:21:07.603936+0000","last_undegraded":"2024-05-03T17:21:07.603936+0000","last_fullsized":"2024-05-03T17:21:07.603936+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:53:48.087190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598616+0000","last_change":"2024-05-03T17:21:07.598616+0000","last_active":"2024-05-03T17:21:07.598616+0000","last_peered":"2024-05-03T17:21:07.598616+0000","last_clean":"2024-05-03T17:21:07.598616+0000","last_became_active":"2024-05-03T17:21:07.598234+0000","last_became_peered":"2024-05-03T17:21:07.598234+0000","last_unstale":"2024-05-03T17:21:07.598616+0000","last_undegraded":"2024-05-03T17:21:07.598616+0000","last_fullsized":"2024-05-03T17:21:07.598616+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:18:59.968590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,10],"acting":[9,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'3","reported_seq":133,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:22:40.730784+0000","last_change":"2024-05-03T17:20:33.172079+0000","last_active":"2024-05-03T17:22:40.730784+0000","last_peered":"2024-05-03T17:22:40.730784+0000","last_clean":"2024-05-03T17:22:40.730784+0000","last_became_active":"2024-05-03T17:20:33.171323+0000","last_became_peered":"2024-05-03T17:20:33.171323+0000","last_unstale":"2024-05-03T17:22:40.730784+0000","last_undegraded":"2024-05-03T17:22:40.730784+0000","last_fullsized":"2024-05-03T17:22:40.730784+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:52:16.091897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":104,"num_read_kb":104,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599057+0000","last_change":"2024-05-03T17:20:37.202340+0000","last_active":"2024-05-03T17:21:07.599057+0000","last_peered":"2024-05-03T17:21:07.599057+0000","last_clean":"2024-05-03T17:21:07.599057+0000","last_became_active":"2024-05-03T17:20:37.201986+0000","last_became_peered":"2024-05-03T17:20:37.201986+0000","last_unstale":"2024-05-03T17:21:07.599057+0000","last_undegraded":"2024-05-03T17:21:07.599057+0000","last_fullsized":"2024-05-03T17:21:07.599057+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:13:47.492653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'2","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.611413+0000","last_change":"2024-05-03T17:21:07.611413+0000","last_active":"2024-05-03T17:21:07.611413+0000","last_peered":"2024-05-03T17:21:07.611413+0000","last_clean":"2024-05-03T17:21:07.611413+0000","last_became_active":"2024-05-03T17:21:07.604448+0000","last_became_peered":"2024-05-03T17:21:07.604448+0000","last_unstale":"2024-05-03T17:21:07.611413+0000","last_undegraded":"2024-05-03T17:21:07.611413+0000","last_fullsized":"2024-05-03T17:21:07.611413+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:28:29.639741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604785+0000","last_change":"2024-05-03T17:21:07.604785+0000","last_active":"2024-05-03T17:21:07.604785+0000","last_peered":"2024-05-03T17:21:07.604785+0000","last_clean":"2024-05-03T17:21:07.604785+0000","last_became_active":"2024-05-03T17:21:07.604507+0000","last_became_peered":"2024-05-03T17:21:07.604507+0000","last_unstale":"2024-05-03T17:21:07.604785+0000","last_undegraded":"2024-05-03T17:21:07.604785+0000","last_fullsized":"2024-05-03T17:21:07.604785+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:53:52.501538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,5],"acting":[0,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604562+0000","last_change":"2024-05-03T17:20:33.284585+0000","last_active":"2024-05-03T17:21:07.604562+0000","last_peered":"2024-05-03T17:21:07.604562+0000","last_clean":"2024-05-03T17:21:07.604562+0000","last_became_active":"2024-05-03T17:20:33.284203+0000","last_became_peered":"2024-05-03T17:20:33.284203+0000","last_unstale":"2024-05-03T17:21:07.604562+0000","last_undegraded":"2024-05-03T17:21:07.604562+0000","last_fullsized":"2024-05-03T17:21:07.604562+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:40:34.608382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598750+0000","last_change":"2024-05-03T17:20:37.203127+0000","last_active":"2024-05-03T17:21:07.598750+0000","last_peered":"2024-05-03T17:21:07.598750+0000","last_clean":"2024-05-03T17:21:07.598750+0000","last_became_active":"2024-05-03T17:20:37.202809+0000","last_became_peered":"2024-05-03T17:20:37.202809+0000","last_unstale":"2024-05-03T17:21:07.598750+0000","last_undegraded":"2024-05-03T17:21:07.598750+0000","last_fullsized":"2024-05-03T17:21:07.598750+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:36:38.136107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'27","reported_seq":269,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:22:41.732153+0000","last_change":"2024-05-03T17:20:33.171487+0000","last_active":"2024-05-03T17:22:41.732153+0000","last_peered":"2024-05-03T17:22:41.732153+0000","last_clean":"2024-05-03T17:22:41.732153+0000","last_became_active":"2024-05-03T17:20:33.171021+0000","last_became_peered":"2024-05-03T17:20:33.171021+0000","last_unstale":"2024-05-03T17:22:41.732153+0000","last_undegraded":"2024-05-03T17:22:41.732153+0000","last_fullsized":"2024-05-03T17:22:41.732153+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:20:05.729189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":51833,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":216,"num_read_kb":158,"num_write":28,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597839+0000","last_change":"2024-05-03T17:20:37.208443+0000","last_active":"2024-05-03T17:21:07.597839+0000","last_peered":"2024-05-03T17:21:07.597839+0000","last_clean":"2024-05-03T17:21:07.597839+0000","last_became_active":"2024-05-03T17:20:37.207869+0000","last_became_peered":"2024-05-03T17:20:37.207869+0000","last_unstale":"2024-05-03T17:21:07.597839+0000","last_undegraded":"2024-05-03T17:21:07.597839+0000","last_fullsized":"2024-05-03T17:21:07.597839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T22:13:28.978735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'6","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594065+0000","last_change":"2024-05-03T17:20:33.167368+0000","last_active":"2024-05-03T17:21:07.594065+0000","last_peered":"2024-05-03T17:21:07.594065+0000","last_clean":"2024-05-03T17:21:07.594065+0000","last_became_active":"2024-05-03T17:20:33.166503+0000","last_became_peered":"2024-05-03T17:20:33.166503+0000","last_unstale":"2024-05-03T17:21:07.594065+0000","last_undegraded":"2024-05-03T17:21:07.594065+0000","last_fullsized":"2024-05-03T17:21:07.594065+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:14:23.150421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595589+0000","last_change":"2024-05-03T17:20:37.201887+0000","last_active":"2024-05-03T17:21:07.595589+0000","last_peered":"2024-05-03T17:21:07.595589+0000","last_clean":"2024-05-03T17:21:07.595589+0000","last_became_active":"2024-05-03T17:20:37.201548+0000","last_became_peered":"2024-05-03T17:20:37.201548+0000","last_unstale":"2024-05-03T17:21:07.595589+0000","last_undegraded":"2024-05-03T17:21:07.595589+0000","last_fullsized":"2024-05-03T17:21:07.595589+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:39:26.416885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604320+0000","last_change":"2024-05-03T17:20:33.168039+0000","last_active":"2024-05-03T17:21:07.604320+0000","last_peered":"2024-05-03T17:21:07.604320+0000","last_clean":"2024-05-03T17:21:07.604320+0000","last_became_active":"2024-05-03T17:20:33.165695+0000","last_became_peered":"2024-05-03T17:20:33.165695+0000","last_unstale":"2024-05-03T17:21:07.604320+0000","last_undegraded":"2024-05-03T17:21:07.604320+0000","last_fullsized":"2024-05-03T17:21:07.604320+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:47:17.964849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596442+0000","last_change":"2024-05-03T17:20:37.207618+0000","last_active":"2024-05-03T17:21:07.596442+0000","last_peered":"2024-05-03T17:21:07.596442+0000","last_clean":"2024-05-03T17:21:07.596442+0000","last_became_active":"2024-05-03T17:20:37.207290+0000","last_became_peered":"2024-05-03T17:20:37.207290+0000","last_unstale":"2024-05-03T17:21:07.596442+0000","last_undegraded":"2024-05-03T17:21:07.596442+0000","last_fullsized":"2024-05-03T17:21:07.596442+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:26:34.085385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'3","reported_seq":136,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:22:41.733004+0000","last_change":"2024-05-03T17:20:33.167328+0000","last_active":"2024-05-03T17:22:41.733004+0000","last_peered":"2024-05-03T17:22:41.733004+0000","last_clean":"2024-05-03T17:22:41.733004+0000","last_became_active":"2024-05-03T17:20:33.166920+0000","last_became_peered":"2024-05-03T17:20:33.166920+0000","last_unstale":"2024-05-03T17:22:41.733004+0000","last_undegraded":"2024-05-03T17:22:41.733004+0000","last_fullsized":"2024-05-03T17:22:41.733004+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T19:04:47.450596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":108,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596215+0000","last_change":"2024-05-03T17:20:37.212691+0000","last_active":"2024-05-03T17:21:07.596215+0000","last_peered":"2024-05-03T17:21:07.596215+0000","last_clean":"2024-05-03T17:21:07.596215+0000","last_became_active":"2024-05-03T17:20:37.208692+0000","last_became_peered":"2024-05-03T17:20:37.208692+0000","last_unstale":"2024-05-03T17:21:07.596215+0000","last_undegraded":"2024-05-03T17:21:07.596215+0000","last_fullsized":"2024-05-03T17:21:07.596215+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:04:29.441208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'15","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597477+0000","last_change":"2024-05-03T17:20:33.285266+0000","last_active":"2024-05-03T17:21:07.597477+0000","last_peered":"2024-05-03T17:21:07.597477+0000","last_clean":"2024-05-03T17:21:07.597477+0000","last_became_active":"2024-05-03T17:20:33.284223+0000","last_became_peered":"2024-05-03T17:20:33.284223+0000","last_unstale":"2024-05-03T17:21:07.597477+0000","last_undegraded":"2024-05-03T17:21:07.597477+0000","last_fullsized":"2024-05-03T17:21:07.597477+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:20:49.077832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604104+0000","last_change":"2024-05-03T17:20:37.205094+0000","last_active":"2024-05-03T17:21:07.604104+0000","last_peered":"2024-05-03T17:21:07.604104+0000","last_clean":"2024-05-03T17:21:07.604104+0000","last_became_active":"2024-05-03T17:20:37.204769+0000","last_became_peered":"2024-05-03T17:20:37.204769+0000","last_unstale":"2024-05-03T17:21:07.604104+0000","last_undegraded":"2024-05-03T17:21:07.604104+0000","last_fullsized":"2024-05-03T17:21:07.604104+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:00:32.129259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598498+0000","last_change":"2024-05-03T17:20:33.167920+0000","last_active":"2024-05-03T17:21:07.598498+0000","last_peered":"2024-05-03T17:21:07.598498+0000","last_clean":"2024-05-03T17:21:07.598498+0000","last_became_active":"2024-05-03T17:20:33.167571+0000","last_became_peered":"2024-05-03T17:20:33.167571+0000","last_unstale":"2024-05-03T17:21:07.598498+0000","last_undegraded":"2024-05-03T17:21:07.598498+0000","last_fullsized":"2024-05-03T17:21:07.598498+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:13:23.462127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595627+0000","last_change":"2024-05-03T17:20:37.208051+0000","last_active":"2024-05-03T17:21:07.595627+0000","last_peered":"2024-05-03T17:21:07.595627+0000","last_clean":"2024-05-03T17:21:07.595627+0000","last_became_active":"2024-05-03T17:20:37.207685+0000","last_became_peered":"2024-05-03T17:20:37.207685+0000","last_unstale":"2024-05-03T17:21:07.595627+0000","last_undegraded":"2024-05-03T17:21:07.595627+0000","last_fullsized":"2024-05-03T17:21:07.595627+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:27:47.685727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593733+0000","last_change":"2024-05-03T17:20:33.285337+0000","last_active":"2024-05-03T17:21:07.593733+0000","last_peered":"2024-05-03T17:21:07.593733+0000","last_clean":"2024-05-03T17:21:07.593733+0000","last_became_active":"2024-05-03T17:20:33.285012+0000","last_became_peered":"2024-05-03T17:20:33.285012+0000","last_unstale":"2024-05-03T17:21:07.593733+0000","last_undegraded":"2024-05-03T17:21:07.593733+0000","last_fullsized":"2024-05-03T17:21:07.593733+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T05:04:44.246674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"81'1","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:08.625595+0000","last_change":"2024-05-03T17:20:37.211216+0000","last_active":"2024-05-03T17:21:08.625595+0000","last_peered":"2024-05-03T17:21:08.625595+0000","last_clean":"2024-05-03T17:21:08.625595+0000","last_became_active":"2024-05-03T17:20:37.210906+0000","last_became_peered":"2024-05-03T17:20:37.210906+0000","last_unstale":"2024-05-03T17:21:08.625595+0000","last_undegraded":"2024-05-03T17:21:08.625595+0000","last_fullsized":"2024-05-03T17:21:08.625595+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:20:45.831133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597079+0000","last_change":"2024-05-03T17:20:33.165952+0000","last_active":"2024-05-03T17:21:07.597079+0000","last_peered":"2024-05-03T17:21:07.597079+0000","last_clean":"2024-05-03T17:21:07.597079+0000","last_became_active":"2024-05-03T17:20:33.165546+0000","last_became_peered":"2024-05-03T17:20:33.165546+0000","last_unstale":"2024-05-03T17:21:07.597079+0000","last_undegraded":"2024-05-03T17:21:07.597079+0000","last_fullsized":"2024-05-03T17:21:07.597079+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:26:44.484883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592830+0000","last_change":"2024-05-03T17:20:37.209107+0000","last_active":"2024-05-03T17:21:07.592830+0000","last_peered":"2024-05-03T17:21:07.592830+0000","last_clean":"2024-05-03T17:21:07.592830+0000","last_became_active":"2024-05-03T17:20:37.208754+0000","last_became_peered":"2024-05-03T17:20:37.208754+0000","last_unstale":"2024-05-03T17:21:07.592830+0000","last_undegraded":"2024-05-03T17:21:07.592830+0000","last_fullsized":"2024-05-03T17:21:07.592830+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:47:22.142530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604315+0000","last_change":"2024-05-03T17:20:37.209092+0000","last_active":"2024-05-03T17:21:07.604315+0000","last_peered":"2024-05-03T17:21:07.604315+0000","last_clean":"2024-05-03T17:21:07.604315+0000","last_became_active":"2024-05-03T17:20:37.208618+0000","last_became_peered":"2024-05-03T17:20:37.208618+0000","last_unstale":"2024-05-03T17:21:07.604315+0000","last_undegraded":"2024-05-03T17:21:07.604315+0000","last_fullsized":"2024-05-03T17:21:07.604315+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:11:09.509930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'4","reported_seq":137,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:22:42.733719+0000","last_change":"2024-05-03T17:20:33.283985+0000","last_active":"2024-05-03T17:22:42.733719+0000","last_peered":"2024-05-03T17:22:42.733719+0000","last_clean":"2024-05-03T17:22:42.733719+0000","last_became_active":"2024-05-03T17:20:33.283629+0000","last_became_peered":"2024-05-03T17:20:33.283629+0000","last_unstale":"2024-05-03T17:22:42.733719+0000","last_undegraded":"2024-05-03T17:22:42.733719+0000","last_fullsized":"2024-05-03T17:22:42.733719+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:03:08.751017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":106,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592649+0000","last_change":"2024-05-03T17:20:37.209086+0000","last_active":"2024-05-03T17:21:07.592649+0000","last_peered":"2024-05-03T17:21:07.592649+0000","last_clean":"2024-05-03T17:21:07.592649+0000","last_became_active":"2024-05-03T17:20:37.207968+0000","last_became_peered":"2024-05-03T17:20:37.207968+0000","last_unstale":"2024-05-03T17:21:07.592649+0000","last_undegraded":"2024-05-03T17:21:07.592649+0000","last_fullsized":"2024-05-03T17:21:07.592649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:57:51.816995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'3","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603078+0000","last_change":"2024-05-03T17:20:33.164302+0000","last_active":"2024-05-03T17:21:07.603078+0000","last_peered":"2024-05-03T17:21:07.603078+0000","last_clean":"2024-05-03T17:21:07.603078+0000","last_became_active":"2024-05-03T17:20:33.163958+0000","last_became_peered":"2024-05-03T17:20:33.163958+0000","last_unstale":"2024-05-03T17:21:07.603078+0000","last_undegraded":"2024-05-03T17:21:07.603078+0000","last_fullsized":"2024-05-03T17:21:07.603078+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:09:08.221659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":101,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596376+0000","last_change":"2024-05-03T17:19:04.188548+0000","last_active":"2024-05-03T17:21:07.596376+0000","last_peered":"2024-05-03T17:21:07.596376+0000","last_clean":"2024-05-03T17:21:07.596376+0000","last_became_active":"2024-05-03T17:19:04.183986+0000","last_became_peered":"2024-05-03T17:19:04.183986+0000","last_unstale":"2024-05-03T17:21:07.596376+0000","last_undegraded":"2024-05-03T17:21:07.596376+0000","last_fullsized":"2024-05-03T17:21:07.596376+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:15:37.276361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:15:37.276361+0000","last_clean_scrub_stamp":"2024-05-03T17:15:37.276361+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:45:57.109987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594869+0000","last_change":"2024-05-03T17:20:37.206686+0000","last_active":"2024-05-03T17:21:07.594869+0000","last_peered":"2024-05-03T17:21:07.594869+0000","last_clean":"2024-05-03T17:21:07.594869+0000","last_became_active":"2024-05-03T17:20:37.204810+0000","last_became_peered":"2024-05-03T17:20:37.204810+0000","last_unstale":"2024-05-03T17:21:07.594869+0000","last_undegraded":"2024-05-03T17:21:07.594869+0000","last_fullsized":"2024-05-03T17:21:07.594869+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:52:11.363372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597296+0000","last_change":"2024-05-03T17:20:33.168412+0000","last_active":"2024-05-03T17:21:07.597296+0000","last_peered":"2024-05-03T17:21:07.597296+0000","last_clean":"2024-05-03T17:21:07.597296+0000","last_became_active":"2024-05-03T17:20:33.168089+0000","last_became_peered":"2024-05-03T17:20:33.168089+0000","last_unstale":"2024-05-03T17:21:07.597296+0000","last_undegraded":"2024-05-03T17:21:07.597296+0000","last_fullsized":"2024-05-03T17:21:07.597296+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:03:11.983138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'1","reported_seq":15,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.602992+0000","last_change":"2024-05-03T17:21:07.602540+0000","last_active":"2024-05-03T17:21:07.602992+0000","last_peered":"2024-05-03T17:21:07.602992+0000","last_clean":"2024-05-03T17:21:07.602992+0000","last_became_active":"2024-05-03T17:21:07.599329+0000","last_became_peered":"2024-05-03T17:21:07.599329+0000","last_unstale":"2024-05-03T17:21:07.602992+0000","last_undegraded":"2024-05-03T17:21:07.602992+0000","last_fullsized":"2024-05-03T17:21:07.602992+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:57:16.367541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594955+0000","last_change":"2024-05-03T17:20:37.206615+0000","last_active":"2024-05-03T17:21:07.594955+0000","last_peered":"2024-05-03T17:21:07.594955+0000","last_clean":"2024-05-03T17:21:07.594955+0000","last_became_active":"2024-05-03T17:20:37.205519+0000","last_became_peered":"2024-05-03T17:20:37.205519+0000","last_unstale":"2024-05-03T17:21:07.594955+0000","last_undegraded":"2024-05-03T17:21:07.594955+0000","last_fullsized":"2024-05-03T17:21:07.594955+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T19:08:45.459749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'11","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:22:37.227095+0000","last_change":"2024-05-03T17:20:33.164933+0000","last_active":"2024-05-03T17:22:37.227095+0000","last_peered":"2024-05-03T17:22:37.227095+0000","last_clean":"2024-05-03T17:22:37.227095+0000","last_became_active":"2024-05-03T17:20:33.164403+0000","last_became_peered":"2024-05-03T17:20:33.164403+0000","last_unstale":"2024-05-03T17:22:37.227095+0000","last_undegraded":"2024-05-03T17:22:37.227095+0000","last_fullsized":"2024-05-03T17:22:37.227095+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:11:21.158371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592505+0000","last_change":"2024-05-03T17:20:37.211145+0000","last_active":"2024-05-03T17:21:07.592505+0000","last_peered":"2024-05-03T17:21:07.592505+0000","last_clean":"2024-05-03T17:21:07.592505+0000","last_became_active":"2024-05-03T17:20:37.210829+0000","last_became_peered":"2024-05-03T17:20:37.210829+0000","last_unstale":"2024-05-03T17:21:07.592505+0000","last_undegraded":"2024-05-03T17:21:07.592505+0000","last_fullsized":"2024-05-03T17:21:07.592505+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:19:23.868001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.603079+0000","last_change":"2024-05-03T17:20:33.166287+0000","last_active":"2024-05-03T17:21:07.603079+0000","last_peered":"2024-05-03T17:21:07.603079+0000","last_clean":"2024-05-03T17:21:07.603079+0000","last_became_active":"2024-05-03T17:20:33.165929+0000","last_became_peered":"2024-05-03T17:20:33.165929+0000","last_unstale":"2024-05-03T17:21:07.603079+0000","last_undegraded":"2024-05-03T17:21:07.603079+0000","last_fullsized":"2024-05-03T17:21:07.603079+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:32:37.093941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599190+0000","last_change":"2024-05-03T17:20:37.186668+0000","last_active":"2024-05-03T17:21:07.599190+0000","last_peered":"2024-05-03T17:21:07.599190+0000","last_clean":"2024-05-03T17:21:07.599190+0000","last_became_active":"2024-05-03T17:20:37.185922+0000","last_became_peered":"2024-05-03T17:20:37.185922+0000","last_unstale":"2024-05-03T17:21:07.599190+0000","last_undegraded":"2024-05-03T17:21:07.599190+0000","last_fullsized":"2024-05-03T17:21:07.599190+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:00:11.075950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598468+0000","last_change":"2024-05-03T17:20:33.284515+0000","last_active":"2024-05-03T17:21:07.598468+0000","last_peered":"2024-05-03T17:21:07.598468+0000","last_clean":"2024-05-03T17:21:07.598468+0000","last_became_active":"2024-05-03T17:20:33.284169+0000","last_became_peered":"2024-05-03T17:20:33.284169+0000","last_unstale":"2024-05-03T17:21:07.598468+0000","last_undegraded":"2024-05-03T17:21:07.598468+0000","last_fullsized":"2024-05-03T17:21:07.598468+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:27:49.722159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597482+0000","last_change":"2024-05-03T17:20:37.200724+0000","last_active":"2024-05-03T17:21:07.597482+0000","last_peered":"2024-05-03T17:21:07.597482+0000","last_clean":"2024-05-03T17:21:07.597482+0000","last_became_active":"2024-05-03T17:20:37.200383+0000","last_became_peered":"2024-05-03T17:20:37.200383+0000","last_unstale":"2024-05-03T17:21:07.597482+0000","last_undegraded":"2024-05-03T17:21:07.597482+0000","last_fullsized":"2024-05-03T17:21:07.597482+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:12:15.547433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592389+0000","last_change":"2024-05-03T17:20:33.163642+0000","last_active":"2024-05-03T17:21:07.592389+0000","last_peered":"2024-05-03T17:21:07.592389+0000","last_clean":"2024-05-03T17:21:07.592389+0000","last_became_active":"2024-05-03T17:20:33.163270+0000","last_became_peered":"2024-05-03T17:20:33.163270+0000","last_unstale":"2024-05-03T17:21:07.592389+0000","last_undegraded":"2024-05-03T17:21:07.592389+0000","last_fullsized":"2024-05-03T17:21:07.592389+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:48:02.719294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595341+0000","last_change":"2024-05-03T17:20:37.203063+0000","last_active":"2024-05-03T17:21:07.595341+0000","last_peered":"2024-05-03T17:21:07.595341+0000","last_clean":"2024-05-03T17:21:07.595341+0000","last_became_active":"2024-05-03T17:20:37.202609+0000","last_became_peered":"2024-05-03T17:20:37.202609+0000","last_unstale":"2024-05-03T17:21:07.595341+0000","last_undegraded":"2024-05-03T17:21:07.595341+0000","last_fullsized":"2024-05-03T17:21:07.595341+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:38:08.497937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593609+0000","last_change":"2024-05-03T17:20:33.168672+0000","last_active":"2024-05-03T17:21:07.593609+0000","last_peered":"2024-05-03T17:21:07.593609+0000","last_clean":"2024-05-03T17:21:07.593609+0000","last_became_active":"2024-05-03T17:20:33.168349+0000","last_became_peered":"2024-05-03T17:20:33.168349+0000","last_unstale":"2024-05-03T17:21:07.593609+0000","last_undegraded":"2024-05-03T17:21:07.593609+0000","last_fullsized":"2024-05-03T17:21:07.593609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:26:04.340040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592378+0000","last_change":"2024-05-03T17:20:37.203725+0000","last_active":"2024-05-03T17:21:07.592378+0000","last_peered":"2024-05-03T17:21:07.592378+0000","last_clean":"2024-05-03T17:21:07.592378+0000","last_became_active":"2024-05-03T17:20:37.203318+0000","last_became_peered":"2024-05-03T17:20:37.203318+0000","last_unstale":"2024-05-03T17:21:07.592378+0000","last_undegraded":"2024-05-03T17:21:07.592378+0000","last_fullsized":"2024-05-03T17:21:07.592378+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T20:00:48.706036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.609823+0000","last_change":"2024-05-03T17:21:07.609392+0000","last_active":"2024-05-03T17:21:07.609823+0000","last_peered":"2024-05-03T17:21:07.609823+0000","last_clean":"2024-05-03T17:21:07.609823+0000","last_became_active":"2024-05-03T17:21:07.600420+0000","last_became_peered":"2024-05-03T17:21:07.600420+0000","last_unstale":"2024-05-03T17:21:07.609823+0000","last_undegraded":"2024-05-03T17:21:07.609823+0000","last_fullsized":"2024-05-03T17:21:07.609823+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:03:47.869058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597082+0000","last_change":"2024-05-03T17:20:37.203766+0000","last_active":"2024-05-03T17:21:07.597082+0000","last_peered":"2024-05-03T17:21:07.597082+0000","last_clean":"2024-05-03T17:21:07.597082+0000","last_became_active":"2024-05-03T17:20:37.203306+0000","last_became_peered":"2024-05-03T17:20:37.203306+0000","last_unstale":"2024-05-03T17:21:07.597082+0000","last_undegraded":"2024-05-03T17:21:07.597082+0000","last_fullsized":"2024-05-03T17:21:07.597082+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:57:22.760234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594059+0000","last_change":"2024-05-03T17:20:33.285014+0000","last_active":"2024-05-03T17:21:07.594059+0000","last_peered":"2024-05-03T17:21:07.594059+0000","last_clean":"2024-05-03T17:21:07.594059+0000","last_became_active":"2024-05-03T17:20:33.284178+0000","last_became_peered":"2024-05-03T17:20:33.284178+0000","last_unstale":"2024-05-03T17:21:07.594059+0000","last_undegraded":"2024-05-03T17:21:07.594059+0000","last_fullsized":"2024-05-03T17:21:07.594059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:45:57.315989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597013+0000","last_change":"2024-05-03T17:20:37.201286+0000","last_active":"2024-05-03T17:21:07.597013+0000","last_peered":"2024-05-03T17:21:07.597013+0000","last_clean":"2024-05-03T17:21:07.597013+0000","last_became_active":"2024-05-03T17:20:37.200879+0000","last_became_peered":"2024-05-03T17:20:37.200879+0000","last_unstale":"2024-05-03T17:21:07.597013+0000","last_undegraded":"2024-05-03T17:21:07.597013+0000","last_fullsized":"2024-05-03T17:21:07.597013+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:25:49.936431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'1","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604390+0000","last_change":"2024-05-03T17:20:33.165850+0000","last_active":"2024-05-03T17:21:07.604390+0000","last_peered":"2024-05-03T17:21:07.604390+0000","last_clean":"2024-05-03T17:21:07.604390+0000","last_became_active":"2024-05-03T17:20:33.165478+0000","last_became_peered":"2024-05-03T17:20:33.165478+0000","last_unstale":"2024-05-03T17:21:07.604390+0000","last_undegraded":"2024-05-03T17:21:07.604390+0000","last_fullsized":"2024-05-03T17:21:07.604390+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T21:22:36.224538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596692+0000","last_change":"2024-05-03T17:20:37.210896+0000","last_active":"2024-05-03T17:21:07.596692+0000","last_peered":"2024-05-03T17:21:07.596692+0000","last_clean":"2024-05-03T17:21:07.596692+0000","last_became_active":"2024-05-03T17:20:37.209150+0000","last_became_peered":"2024-05-03T17:20:37.209150+0000","last_unstale":"2024-05-03T17:21:07.596692+0000","last_undegraded":"2024-05-03T17:21:07.596692+0000","last_fullsized":"2024-05-03T17:21:07.596692+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:17:01.870395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594502+0000","last_change":"2024-05-03T17:20:33.169631+0000","last_active":"2024-05-03T17:21:07.594502+0000","last_peered":"2024-05-03T17:21:07.594502+0000","last_clean":"2024-05-03T17:21:07.594502+0000","last_became_active":"2024-05-03T17:20:33.169293+0000","last_became_peered":"2024-05-03T17:20:33.169293+0000","last_unstale":"2024-05-03T17:21:07.594502+0000","last_undegraded":"2024-05-03T17:21:07.594502+0000","last_fullsized":"2024-05-03T17:21:07.594502+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:32.147004+0000","last_clean_scrub_stamp":"2024-05-03T17:20:32.147004+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:20:29.988150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.597243+0000","last_change":"2024-05-03T17:20:37.201347+0000","last_active":"2024-05-03T17:21:07.597243+0000","last_peered":"2024-05-03T17:21:07.597243+0000","last_clean":"2024-05-03T17:21:07.597243+0000","last_became_active":"2024-05-03T17:20:37.200999+0000","last_became_peered":"2024-05-03T17:20:37.200999+0000","last_unstale":"2024-05-03T17:21:07.597243+0000","last_undegraded":"2024-05-03T17:21:07.597243+0000","last_fullsized":"2024-05-03T17:21:07.597243+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:34:23.027957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595391+0000","last_change":"2024-05-03T17:20:37.206379+0000","last_active":"2024-05-03T17:21:07.595391+0000","last_peered":"2024-05-03T17:21:07.595391+0000","last_clean":"2024-05-03T17:21:07.595391+0000","last_became_active":"2024-05-03T17:20:37.206048+0000","last_became_peered":"2024-05-03T17:20:37.206048+0000","last_unstale":"2024-05-03T17:21:07.595391+0000","last_undegraded":"2024-05-03T17:21:07.595391+0000","last_fullsized":"2024-05-03T17:21:07.595391+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T01:46:02.486170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593822+0000","last_change":"2024-05-03T17:20:37.205997+0000","last_active":"2024-05-03T17:21:07.593822+0000","last_peered":"2024-05-03T17:21:07.593822+0000","last_clean":"2024-05-03T17:21:07.593822+0000","last_became_active":"2024-05-03T17:20:37.205602+0000","last_became_peered":"2024-05-03T17:20:37.205602+0000","last_unstale":"2024-05-03T17:21:07.593822+0000","last_undegraded":"2024-05-03T17:21:07.593822+0000","last_fullsized":"2024-05-03T17:21:07.593822+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:16:04.929294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.593673+0000","last_change":"2024-05-03T17:20:37.199994+0000","last_active":"2024-05-03T17:21:07.593673+0000","last_peered":"2024-05-03T17:21:07.593673+0000","last_clean":"2024-05-03T17:21:07.593673+0000","last_became_active":"2024-05-03T17:20:37.199653+0000","last_became_peered":"2024-05-03T17:20:37.199653+0000","last_unstale":"2024-05-03T17:21:07.593673+0000","last_undegraded":"2024-05-03T17:21:07.593673+0000","last_fullsized":"2024-05-03T17:21:07.593673+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T04:23:24.945036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598045+0000","last_change":"2024-05-03T17:20:37.198701+0000","last_active":"2024-05-03T17:21:07.598045+0000","last_peered":"2024-05-03T17:21:07.598045+0000","last_clean":"2024-05-03T17:21:07.598045+0000","last_became_active":"2024-05-03T17:20:37.198374+0000","last_became_peered":"2024-05-03T17:20:37.198374+0000","last_unstale":"2024-05-03T17:21:07.598045+0000","last_undegraded":"2024-05-03T17:21:07.598045+0000","last_fullsized":"2024-05-03T17:21:07.598045+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T18:33:26.744746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.599746+0000","last_change":"2024-05-03T17:21:07.599746+0000","last_active":"2024-05-03T17:21:07.599746+0000","last_peered":"2024-05-03T17:21:07.599746+0000","last_clean":"2024-05-03T17:21:07.599746+0000","last_became_active":"2024-05-03T17:21:07.598862+0000","last_became_peered":"2024-05-03T17:21:07.598862+0000","last_unstale":"2024-05-03T17:21:07.599746+0000","last_undegraded":"2024-05-03T17:21:07.599746+0000","last_fullsized":"2024-05-03T17:21:07.599746+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T02:46:31.214548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,9,0],"acting":[6,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.596253+0000","last_change":"2024-05-03T17:20:37.207194+0000","last_active":"2024-05-03T17:21:07.596253+0000","last_peered":"2024-05-03T17:21:07.596253+0000","last_clean":"2024-05-03T17:21:07.596253+0000","last_became_active":"2024-05-03T17:20:37.206833+0000","last_became_peered":"2024-05-03T17:20:37.206833+0000","last_unstale":"2024-05-03T17:21:07.596253+0000","last_undegraded":"2024-05-03T17:21:07.596253+0000","last_fullsized":"2024-05-03T17:21:07.596253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:29:32.212841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.594069+0000","last_change":"2024-05-03T17:20:37.203613+0000","last_active":"2024-05-03T17:21:07.594069+0000","last_peered":"2024-05-03T17:21:07.594069+0000","last_clean":"2024-05-03T17:21:07.594069+0000","last_became_active":"2024-05-03T17:20:37.203265+0000","last_became_peered":"2024-05-03T17:20:37.203265+0000","last_unstale":"2024-05-03T17:21:07.594069+0000","last_undegraded":"2024-05-03T17:21:07.594069+0000","last_fullsized":"2024-05-03T17:21:07.594069+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:12:35.816804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592409+0000","last_change":"2024-05-03T17:20:37.206208+0000","last_active":"2024-05-03T17:21:07.592409+0000","last_peered":"2024-05-03T17:21:07.592409+0000","last_clean":"2024-05-03T17:21:07.592409+0000","last_became_active":"2024-05-03T17:20:37.205859+0000","last_became_peered":"2024-05-03T17:20:37.205859+0000","last_unstale":"2024-05-03T17:21:07.592409+0000","last_undegraded":"2024-05-03T17:21:07.592409+0000","last_fullsized":"2024-05-03T17:21:07.592409+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T23:57:55.311154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.604102+0000","last_change":"2024-05-03T17:20:37.197239+0000","last_active":"2024-05-03T17:21:07.604102+0000","last_peered":"2024-05-03T17:21:07.604102+0000","last_clean":"2024-05-03T17:21:07.604102+0000","last_became_active":"2024-05-03T17:20:37.196903+0000","last_became_peered":"2024-05-03T17:20:37.196903+0000","last_unstale":"2024-05-03T17:21:07.604102+0000","last_undegraded":"2024-05-03T17:21:07.604102+0000","last_fullsized":"2024-05-03T17:21:07.604102+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:29:53.020573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.592086+0000","last_change":"2024-05-03T17:20:37.211295+0000","last_active":"2024-05-03T17:21:07.592086+0000","last_peered":"2024-05-03T17:21:07.592086+0000","last_clean":"2024-05-03T17:21:07.592086+0000","last_became_active":"2024-05-03T17:20:37.210939+0000","last_became_peered":"2024-05-03T17:20:37.210939+0000","last_unstale":"2024-05-03T17:21:07.592086+0000","last_undegraded":"2024-05-03T17:21:07.592086+0000","last_fullsized":"2024-05-03T17:21:07.592086+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-04T17:45:58.770190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.598703+0000","last_change":"2024-05-03T17:20:37.209581+0000","last_active":"2024-05-03T17:21:07.598703+0000","last_peered":"2024-05-03T17:21:07.598703+0000","last_clean":"2024-05-03T17:21:07.598703+0000","last_became_active":"2024-05-03T17:20:37.209126+0000","last_became_peered":"2024-05-03T17:20:37.209126+0000","last_unstale":"2024-05-03T17:21:07.598703+0000","last_undegraded":"2024-05-03T17:21:07.598703+0000","last_fullsized":"2024-05-03T17:21:07.598703+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:47:15.485725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595388+0000","last_change":"2024-05-03T17:20:37.204258+0000","last_active":"2024-05-03T17:21:07.595388+0000","last_peered":"2024-05-03T17:21:07.595388+0000","last_clean":"2024-05-03T17:21:07.595388+0000","last_became_active":"2024-05-03T17:20:37.202405+0000","last_became_peered":"2024-05-03T17:20:37.202405+0000","last_unstale":"2024-05-03T17:21:07.595388+0000","last_undegraded":"2024-05-03T17:21:07.595388+0000","last_fullsized":"2024-05-03T17:21:07.595388+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T00:36:35.946607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-03T17:21:07.595536+0000","last_change":"2024-05-03T17:20:37.208909+0000","last_active":"2024-05-03T17:21:07.595536+0000","last_peered":"2024-05-03T17:21:07.595536+0000","last_clean":"2024-05-03T17:21:07.595536+0000","last_became_active":"2024-05-03T17:20:37.208540+0000","last_became_peered":"2024-05-03T17:20:37.208540+0000","last_unstale":"2024-05-03T17:21:07.595536+0000","last_undegraded":"2024-05-03T17:21:07.595536+0000","last_fullsized":"2024-05-03T17:21:07.595536+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-03T17:20:36.168128+0000","last_clean_scrub_stamp":"2024-05-03T17:20:36.168128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-05T03:32:41.700487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":213988,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":55834,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":651,"num_read_kb":384,"num_write":162,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":167502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678058,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":876544,"data_stored":428314,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874287,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36328,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708824,"statfs":{"total":95995035648,"available":95957835776,"internally_reserved":0,"allocated":1015808,"data_stored":1071606,"data_compressed":42001,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037813,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":428528,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234042,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":428426,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430271,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":1020378,"data_compressed":42001,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593797,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32040,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713112,"statfs":{"total":95995035648,"available":95962226688,"internally_reserved":0,"allocated":876544,"data_stored":427917,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":479543,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32056,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713096,"statfs":{"total":95995035648,"available":95962210304,"internally_reserved":0,"allocated":897024,"data_stored":429188,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":481284,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":892928,"data_stored":428228,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953551,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":880640,"data_stored":428354,"data_compressed":5795,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790026,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":1020062,"data_compressed":42001,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34582,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34582,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36206,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":51833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":53574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":53411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-03T17:22:46.408 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-03T17:22:46.408 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-03T17:22:46.409 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-03T17:22:46.409 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph health --format=json 2024-05-03T17:22:46.661 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:46.924 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:46 smithi067 ceph-mon[29731]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:22:47.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:46 smithi082 ceph-mon[33535]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:22:47.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:46 smithi070 ceph-mon[33250]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-03T17:22:47.790 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:47 smithi067 ceph-mon[29731]: pgmap v361: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:22:48.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:47 smithi082 ceph-mon[33535]: pgmap v361: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:22:48.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:47 smithi070 ceph-mon[33250]: pgmap v361: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:22:48.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.315+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/2980641397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6dc102fd0 msgr2=0x7ff6dc107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:48.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.315+0000 7ff6e0ba4700 1 --2- 172.21.15.67:0/2980641397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6dc102fd0 0x7ff6dc107020 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ff6d00099c0 tx=0x7ff6d0009cd0 comp rx=0 tx=0).stop 2024-05-03T17:22:48.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.315+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/2980641397 shutdown_connections 2024-05-03T17:22:48.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.315+0000 7ff6e0ba4700 1 --2- 172.21.15.67:0/2980641397 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6dc107700 0x7ff6dc10f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:48.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.315+0000 7ff6e0ba4700 1 --2- 172.21.15.67:0/2980641397 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6dc102620 0x7ff6dc102a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:48.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.315+0000 7ff6e0ba4700 1 --2- 172.21.15.67:0/2980641397 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6dc102fd0 0x7ff6dc107020 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:48.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.315+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/2980641397 >> 172.21.15.67:0/2980641397 conn(0x7ff6dc079440 msgr2=0x7ff6dc079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:48.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.315+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/2980641397 shutdown_connections 2024-05-03T17:22:48.316 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.315+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/2980641397 wait complete. 2024-05-03T17:22:48.318 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6e0ba4700 1 Processor -- start 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6e0ba4700 1 -- start start 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6e0ba4700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6dc102620 0x7ff6dc19a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6e0ba4700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6dc102fd0 0x7ff6dc19af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6e0ba4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6dc107700 0x7ff6dc1a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6e0ba4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc112570 con 0x7ff6dc107700 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6e0ba4700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc1123f0 con 0x7ff6dc102620 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6e0ba4700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff6dc1126f0 con 0x7ff6dc102fd0 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6dad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6dc107700 0x7ff6dc1a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6dad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6dc107700 0x7ff6dc1a5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53622/0 (socket says 172.21.15.67:53622) 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6dad9d700 1 -- 172.21.15.67:0/226812203 learned_addr learned my addr 172.21.15.67:0/226812203 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6d9d9b700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6dc102fd0 0x7ff6dc19af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.316+0000 7ff6da59c700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6dc102620 0x7ff6dc19a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.317+0000 7ff6dad9d700 1 -- 172.21.15.67:0/226812203 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6dc102fd0 msgr2=0x7ff6dc19af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.317+0000 7ff6dad9d700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6dc102fd0 0x7ff6dc19af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.317+0000 7ff6dad9d700 1 -- 172.21.15.67:0/226812203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6dc102620 msgr2=0x7ff6dc19a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:48.319 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.317+0000 7ff6dad9d700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6dc102620 0x7ff6dc19a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.317+0000 7ff6dad9d700 1 -- 172.21.15.67:0/226812203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6dc1a5fb0 con 0x7ff6dc107700 2024-05-03T17:22:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.317+0000 7ff6dad9d700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6dc107700 0x7ff6dc1a5a70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400ef30 tx=0x7ff6d400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.317+0000 7ff6cb7fe700 1 -- 172.21.15.67:0/226812203 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d40092e0 con 0x7ff6dc107700 2024-05-03T17:22:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.317+0000 7ff6cb7fe700 1 -- 172.21.15.67:0/226812203 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6d4004bb0 con 0x7ff6dc107700 2024-05-03T17:22:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.317+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/226812203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6dc1a6180 con 0x7ff6dc107700 2024-05-03T17:22:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.317+0000 7ff6cb7fe700 1 -- 172.21.15.67:0/226812203 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4004280 con 0x7ff6dc107700 2024-05-03T17:22:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.317+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/226812203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6dc1a66e0 con 0x7ff6dc107700 2024-05-03T17:22:48.320 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.318+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/226812203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6dc06a190 con 0x7ff6dc107700 2024-05-03T17:22:48.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.319+0000 7ff6cb7fe700 1 -- 172.21.15.67:0/226812203 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff6d401b030 con 0x7ff6dc107700 2024-05-03T17:22:48.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.320+0000 7ff6cb7fe700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff6c40b2990 0x7ff6c40b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:48.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.320+0000 7ff6da59c700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff6c40b2990 0x7ff6c40b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:48.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.320+0000 7ff6cb7fe700 1 -- 172.21.15.67:0/226812203 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7ff6d40e2800 con 0x7ff6dc107700 2024-05-03T17:22:48.321 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.321+0000 7ff6da59c700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff6c40b2990 0x7ff6c40b4e50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc005fd0 tx=0x7ff6cc005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:48.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.324+0000 7ff6cb7fe700 1 -- 172.21.15.67:0/226812203 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ff6d4022020 con 0x7ff6dc107700 2024-05-03T17:22:48.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.588+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/226812203 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff6dc02e510 con 0x7ff6dc107700 2024-05-03T17:22:48.589 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.588+0000 7ff6cb7fe700 1 -- 172.21.15.67:0/226812203 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7ff6d4010040 con 0x7ff6dc107700 2024-05-03T17:22:48.589 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:22:48.589 INFO:teuthology.orchestra.run.smithi067.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-03T17:22:48.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.590+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/226812203 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff6c40b2990 msgr2=0x7ff6c40b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:48.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.590+0000 7ff6e0ba4700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff6c40b2990 0x7ff6c40b4e50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc005fd0 tx=0x7ff6cc005f60 comp rx=0 tx=0).stop 2024-05-03T17:22:48.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.590+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/226812203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6dc107700 msgr2=0x7ff6dc1a5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:48.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.590+0000 7ff6e0ba4700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6dc107700 0x7ff6dc1a5a70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400ef30 tx=0x7ff6d400c3a0 comp rx=0 tx=0).stop 2024-05-03T17:22:48.591 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.591+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/226812203 shutdown_connections 2024-05-03T17:22:48.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.591+0000 7ff6e0ba4700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff6c40b2990 0x7ff6c40b4e50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:48.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.591+0000 7ff6e0ba4700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff6dc107700 0x7ff6dc1a5a70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:48.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.591+0000 7ff6e0ba4700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff6dc102620 0x7ff6dc19a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:48.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.591+0000 7ff6e0ba4700 1 --2- 172.21.15.67:0/226812203 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff6dc102fd0 0x7ff6dc19af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:48.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.591+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/226812203 >> 172.21.15.67:0/226812203 conn(0x7ff6dc079440 msgr2=0x7ff6dc1005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:48.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.591+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/226812203 shutdown_connections 2024-05-03T17:22:48.592 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:48.591+0000 7ff6e0ba4700 1 -- 172.21.15.67:0/226812203 wait complete. 2024-05-03T17:22:48.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:48 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/226812203' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-03T17:22:49.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:48 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/226812203' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-03T17:22:49.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:48 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/226812203' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-03T17:22:49.336 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-03T17:22:49.337 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:22:49.588 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:49.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:49 smithi067 ceph-mon[29731]: pgmap v362: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-03T17:22:50.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:49 smithi082 ceph-mon[33535]: pgmap v362: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-03T17:22:50.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:49 smithi070 ceph-mon[33250]: pgmap v362: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-03T17:22:51.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.275+0000 7f4150eb7700 1 -- 172.21.15.67:0/2210550318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f414c108850 msgr2=0x7f414c1103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:51.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.275+0000 7f4150eb7700 1 --2- 172.21.15.67:0/2210550318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f414c108850 0x7f414c1103d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f413c0099c0 tx=0x7f413c009cd0 comp rx=0 tx=0).stop 2024-05-03T17:22:51.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.275+0000 7f4150eb7700 1 -- 172.21.15.67:0/2210550318 shutdown_connections 2024-05-03T17:22:51.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.275+0000 7f4150eb7700 1 --2- 172.21.15.67:0/2210550318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f414c108850 0x7f414c1103d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:51.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.275+0000 7f4150eb7700 1 --2- 172.21.15.67:0/2210550318 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f414c103750 0x7f414c103b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:51.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.275+0000 7f4150eb7700 1 --2- 172.21.15.67:0/2210550318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f414c104100 0x7f414c108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:51.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.275+0000 7f4150eb7700 1 -- 172.21.15.67:0/2210550318 >> 172.21.15.67:0/2210550318 conn(0x7f414c07a600 msgr2=0x7f414c07aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:51.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.275+0000 7f4150eb7700 1 -- 172.21.15.67:0/2210550318 shutdown_connections 2024-05-03T17:22:51.276 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.275+0000 7f4150eb7700 1 -- 172.21.15.67:0/2210550318 wait complete. 2024-05-03T17:22:51.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f4150eb7700 1 Processor -- start 2024-05-03T17:22:51.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f4150eb7700 1 -- start start 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f4150eb7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f414c103750 0x7f414c0757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f4150eb7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f414c104100 0x7f414c075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f4150eb7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f414c108850 0x7f414c076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f4150eb7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f414c113640 con 0x7f414c108850 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f4150eb7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f414c1134c0 con 0x7f414c103750 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f4150eb7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f414c1137c0 con 0x7f414c104100 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f414ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f414c108850 0x7f414c076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f414ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f414c108850 0x7f414c076220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53638/0 (socket says 172.21.15.67:53638) 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f414ad9d700 1 -- 172.21.15.67:0/4185584402 learned_addr learned my addr 172.21.15.67:0/4185584402 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f414a59c700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f414c103750 0x7f414c0757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f4149d9b700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f414c104100 0x7f414c075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f414ad9d700 1 -- 172.21.15.67:0/4185584402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f414c104100 msgr2=0x7f414c075ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f414ad9d700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f414c104100 0x7f414c075ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f414ad9d700 1 -- 172.21.15.67:0/4185584402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f414c103750 msgr2=0x7f414c0757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:51.279 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f414ad9d700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f414c103750 0x7f414c0757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:51.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f414ad9d700 1 -- 172.21.15.67:0/4185584402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f414c1a7f60 con 0x7f414c108850 2024-05-03T17:22:51.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.276+0000 7f414a59c700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f414c103750 0x7f414c0757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:22:51.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.277+0000 7f414ad9d700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f414c108850 0x7f414c076220 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f413c005910 tx=0x7f413c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:51.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.277+0000 7f41437fe700 1 -- 172.21.15.67:0/4185584402 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c027070 con 0x7f414c108850 2024-05-03T17:22:51.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.277+0000 7f4150eb7700 1 -- 172.21.15.67:0/4185584402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f414c1a81f0 con 0x7f414c108850 2024-05-03T17:22:51.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.277+0000 7f41437fe700 1 -- 172.21.15.67:0/4185584402 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f413c003fa0 con 0x7f414c108850 2024-05-03T17:22:51.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.277+0000 7f41437fe700 1 -- 172.21.15.67:0/4185584402 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f413c022d10 con 0x7f414c108850 2024-05-03T17:22:51.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.277+0000 7f4150eb7700 1 -- 172.21.15.67:0/4185584402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f414c1a86a0 con 0x7f414c108850 2024-05-03T17:22:51.280 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.278+0000 7f4150eb7700 1 -- 172.21.15.67:0/4185584402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f414c10ba20 con 0x7f414c108850 2024-05-03T17:22:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.279+0000 7f41437fe700 1 -- 172.21.15.67:0/4185584402 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f413c02a070 con 0x7f414c108850 2024-05-03T17:22:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.280+0000 7f41437fe700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f41380b2910 0x7f41380b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.280+0000 7f414a59c700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f41380b2910 0x7f41380b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.280+0000 7f41437fe700 1 -- 172.21.15.67:0/4185584402 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f413c0e3230 con 0x7f414c108850 2024-05-03T17:22:51.281 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.280+0000 7f414a59c700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f41380b2910 0x7f41380b4dd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4144009dd0 tx=0x7f4144009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:51.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.283+0000 7f41437fe700 1 -- 172.21.15.67:0/4185584402 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f413c013070 con 0x7f414c108850 2024-05-03T17:22:51.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.544+0000 7f4150eb7700 1 -- 172.21.15.67:0/4185584402 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f414c076cd0 con 0x7f414c108850 2024-05-03T17:22:51.546 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.545+0000 7f41437fe700 1 -- 172.21.15.67:0/4185584402 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11850 (secure 0 0 0) 0x7f413c0aae70 con 0x7f414c108850 2024-05-03T17:22:51.546 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:22:51.546 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/316463706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":316463706},{"type":"v1","addr":"172.21.15.67:6839","nonce":316463706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3422086111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3422086111},{"type":"v1","addr":"172.21.15.67:6841","nonce":3422086111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T17:20:39.322107+0000","modified":"2024-05-03T17:21:01.753515+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24464,"mds_1":24446,"mds_2":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24421":{"gid":24421,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1568454842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1568454842},{"type":"v1","addr":"172.21.15.82:6835","nonce":1568454842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/370282857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":370282857},{"type":"v1","addr":"172.21.15.82:6837","nonce":370282857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.82:6839/2084218047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2084218047},{"type":"v1","addr":"172.21.15.82:6839","nonce":2084218047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.70:6837/3323713018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3323713018},{"type":"v1","addr":"172.21.15.70:6837","nonce":3323713018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6839/902276547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":902276547},{"type":"v1","addr":"172.21.15.70:6839","nonce":902276547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24446,24464,24442]},"id":1}]} 2024-05-03T17:22:51.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.548+0000 7f4150eb7700 1 -- 172.21.15.67:0/4185584402 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f41380b2910 msgr2=0x7f41380b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:51.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.548+0000 7f4150eb7700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f41380b2910 0x7f41380b4dd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4144009dd0 tx=0x7f4144009040 comp rx=0 tx=0).stop 2024-05-03T17:22:51.548 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.548+0000 7f4150eb7700 1 -- 172.21.15.67:0/4185584402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f414c108850 msgr2=0x7f414c076220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:51.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.548+0000 7f4150eb7700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f414c108850 0x7f414c076220 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f413c005910 tx=0x7f413c009580 comp rx=0 tx=0).stop 2024-05-03T17:22:51.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.548+0000 7f4150eb7700 1 -- 172.21.15.67:0/4185584402 shutdown_connections 2024-05-03T17:22:51.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.548+0000 7f4150eb7700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f41380b2910 0x7f41380b4dd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:51.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.548+0000 7f4150eb7700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f414c108850 0x7f414c076220 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:51.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.548+0000 7f4150eb7700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f414c103750 0x7f414c0757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:51.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.548+0000 7f4150eb7700 1 --2- 172.21.15.67:0/4185584402 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f414c104100 0x7f414c075ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:51.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.548+0000 7f4150eb7700 1 -- 172.21.15.67:0/4185584402 >> 172.21.15.67:0/4185584402 conn(0x7f414c07a600 msgr2=0x7f414c1099a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:51.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.548+0000 7f4150eb7700 1 -- 172.21.15.67:0/4185584402 shutdown_connections 2024-05-03T17:22:51.549 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:51.548+0000 7f4150eb7700 1 -- 172.21.15.67:0/4185584402 wait complete. 2024-05-03T17:22:51.549 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T17:22:51.866 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:51 smithi067 ceph-mon[29731]: pgmap v363: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:22:51.867 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:51 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/4185584402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:22:52.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:51 smithi082 ceph-mon[33535]: pgmap v363: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:22:52.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:51 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/4185584402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:22:52.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:51 smithi070 ceph-mon[33250]: pgmap v363: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:22:52.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:51 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/4185584402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:22:52.264 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:22:52.514 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:53.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:53 smithi067 ceph-mon[29731]: pgmap v364: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:22:54.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:53 smithi082 ceph-mon[33535]: pgmap v364: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:22:54.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:53 smithi070 ceph-mon[33250]: pgmap v364: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:22:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.212+0000 7f89e469a700 1 -- 172.21.15.67:0/1302427510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc103540 msgr2=0x7f89dc1039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.212+0000 7f89e469a700 1 --2- 172.21.15.67:0/1302427510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc103540 0x7f89dc1039c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f89d80099c0 tx=0x7f89d8009cd0 comp rx=0 tx=0).stop 2024-05-03T17:22:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.212+0000 7f89e469a700 1 -- 172.21.15.67:0/1302427510 shutdown_connections 2024-05-03T17:22:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.212+0000 7f89e469a700 1 --2- 172.21.15.67:0/1302427510 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89dc103f00 0x7f89dc1103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.212+0000 7f89e469a700 1 --2- 172.21.15.67:0/1302427510 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89dc1095e0 0x7f89dc1099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.212+0000 7f89e469a700 1 --2- 172.21.15.67:0/1302427510 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc103540 0x7f89dc1039c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.212+0000 7f89e469a700 1 -- 172.21.15.67:0/1302427510 >> 172.21.15.67:0/1302427510 conn(0x7f89dc07a650 msgr2=0x7f89dc07aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.212+0000 7f89e469a700 1 -- 172.21.15.67:0/1302427510 shutdown_connections 2024-05-03T17:22:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.212+0000 7f89e469a700 1 -- 172.21.15.67:0/1302427510 wait complete. 2024-05-03T17:22:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e469a700 1 Processor -- start 2024-05-03T17:22:54.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e469a700 1 -- start start 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e469a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89dc103540 0x7f89dc0757a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e469a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc103f00 0x7f89dc075ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e469a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89dc1095e0 0x7f89dc076220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e469a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc113790 con 0x7f89dc103f00 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e469a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc113610 con 0x7f89dc1095e0 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e469a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f89dc113910 con 0x7f89dc103540 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e1c35700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc103f00 0x7f89dc075ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e2436700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89dc103540 0x7f89dc0757a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e2436700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89dc103540 0x7f89dc0757a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:42500/0 (socket says 172.21.15.67:42500) 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e1c35700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc103f00 0x7f89dc075ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53646/0 (socket says 172.21.15.67:53646) 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e2436700 1 -- 172.21.15.67:0/2000890503 learned_addr learned my addr 172.21.15.67:0/2000890503 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.213+0000 7f89e2c37700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89dc1095e0 0x7f89dc076220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.214+0000 7f89e1c35700 1 -- 172.21.15.67:0/2000890503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89dc103540 msgr2=0x7f89dc0757a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.214+0000 7f89e1c35700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89dc103540 0x7f89dc0757a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:54.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.214+0000 7f89e1c35700 1 -- 172.21.15.67:0/2000890503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89dc1095e0 msgr2=0x7f89dc076220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:54.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.214+0000 7f89e1c35700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89dc1095e0 0x7f89dc076220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:54.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.214+0000 7f89e1c35700 1 -- 172.21.15.67:0/2000890503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89dc1a8110 con 0x7f89dc103f00 2024-05-03T17:22:54.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.214+0000 7f89e2436700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89dc103540 0x7f89dc0757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:22:54.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.214+0000 7f89e1c35700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc103f00 0x7f89dc075ce0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009f50 tx=0x7f89d80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:54.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.214+0000 7f89cf7fe700 1 -- 172.21.15.67:0/2000890503 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8027070 con 0x7f89dc103f00 2024-05-03T17:22:54.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.214+0000 7f89cf7fe700 1 -- 172.21.15.67:0/2000890503 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f89d8003f20 con 0x7f89dc103f00 2024-05-03T17:22:54.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.214+0000 7f89e469a700 1 -- 172.21.15.67:0/2000890503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89dc1a83a0 con 0x7f89dc103f00 2024-05-03T17:22:54.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.214+0000 7f89cf7fe700 1 -- 172.21.15.67:0/2000890503 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d8022bd0 con 0x7f89dc103f00 2024-05-03T17:22:54.217 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.214+0000 7f89e469a700 1 -- 172.21.15.67:0/2000890503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89dc1a8850 con 0x7f89dc103f00 2024-05-03T17:22:54.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.215+0000 7f89e469a700 1 -- 172.21.15.67:0/2000890503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89dc104c50 con 0x7f89dc103f00 2024-05-03T17:22:54.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.216+0000 7f89cf7fe700 1 -- 172.21.15.67:0/2000890503 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f89d802a070 con 0x7f89dc103f00 2024-05-03T17:22:54.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.217+0000 7f89cf7fe700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f89c80b2990 0x7f89c80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:54.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.217+0000 7f89e2436700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f89c80b2990 0x7f89c80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:54.218 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.217+0000 7f89cf7fe700 1 -- 172.21.15.67:0/2000890503 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f89d80e30f0 con 0x7f89dc103f00 2024-05-03T17:22:54.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.218+0000 7f89e2436700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f89c80b2990 0x7f89c80b4e50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f89d00104e0 tx=0x7f89d0007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:54.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.226+0000 7f89cf7fe700 1 -- 172.21.15.67:0/2000890503 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f89d8013070 con 0x7f89dc103f00 2024-05-03T17:22:54.493 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.493+0000 7f89e469a700 1 -- 172.21.15.67:0/2000890503 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f89dc02e510 con 0x7f89dc103f00 2024-05-03T17:22:54.494 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.493+0000 7f89cf7fe700 1 -- 172.21.15.67:0/2000890503 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11850 (secure 0 0 0) 0x7f89d80aad30 con 0x7f89dc103f00 2024-05-03T17:22:54.494 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:22:54.494 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/316463706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":316463706},{"type":"v1","addr":"172.21.15.67:6839","nonce":316463706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3422086111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3422086111},{"type":"v1","addr":"172.21.15.67:6841","nonce":3422086111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T17:20:39.322107+0000","modified":"2024-05-03T17:21:01.753515+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24464,"mds_1":24446,"mds_2":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24421":{"gid":24421,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1568454842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1568454842},{"type":"v1","addr":"172.21.15.82:6835","nonce":1568454842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/370282857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":370282857},{"type":"v1","addr":"172.21.15.82:6837","nonce":370282857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.82:6839/2084218047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2084218047},{"type":"v1","addr":"172.21.15.82:6839","nonce":2084218047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.70:6837/3323713018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3323713018},{"type":"v1","addr":"172.21.15.70:6837","nonce":3323713018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6839/902276547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":902276547},{"type":"v1","addr":"172.21.15.70:6839","nonce":902276547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24446,24464,24442]},"id":1}]} 2024-05-03T17:22:54.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.497+0000 7f89e469a700 1 -- 172.21.15.67:0/2000890503 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f89c80b2990 msgr2=0x7f89c80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:54.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.497+0000 7f89e469a700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f89c80b2990 0x7f89c80b4e50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f89d00104e0 tx=0x7f89d0007480 comp rx=0 tx=0).stop 2024-05-03T17:22:54.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.497+0000 7f89e469a700 1 -- 172.21.15.67:0/2000890503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc103f00 msgr2=0x7f89dc075ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:54.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.497+0000 7f89e469a700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc103f00 0x7f89dc075ce0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009f50 tx=0x7f89d80047b0 comp rx=0 tx=0).stop 2024-05-03T17:22:54.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.497+0000 7f89e469a700 1 -- 172.21.15.67:0/2000890503 shutdown_connections 2024-05-03T17:22:54.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.497+0000 7f89e469a700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f89c80b2990 0x7f89c80b4e50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:54.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.497+0000 7f89e469a700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f89dc1095e0 0x7f89dc076220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:54.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.497+0000 7f89e469a700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f89dc103540 0x7f89dc0757a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:54.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.497+0000 7f89e469a700 1 --2- 172.21.15.67:0/2000890503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f89dc103f00 0x7f89dc075ce0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:54.498 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.497+0000 7f89e469a700 1 -- 172.21.15.67:0/2000890503 >> 172.21.15.67:0/2000890503 conn(0x7f89dc07a650 msgr2=0x7f89dc102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:54.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.497+0000 7f89e469a700 1 -- 172.21.15.67:0/2000890503 shutdown_connections 2024-05-03T17:22:54.499 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:54.497+0000 7f89e469a700 1 -- 172.21.15.67:0/2000890503 wait complete. 2024-05-03T17:22:54.499 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T17:22:54.884 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:54 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2000890503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:22:55.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:54 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2000890503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:22:55.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:54 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2000890503' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:22:55.229 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T17:22:55.478 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:55.851 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:55 smithi067 ceph-mon[29731]: pgmap v365: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:22:56.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:55 smithi082 ceph-mon[33535]: pgmap v365: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:22:56.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:55 smithi070 ceph-mon[33250]: pgmap v365: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:22:57.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.179+0000 7fe84a0b3700 1 -- 172.21.15.67:0/1197003885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8441095d0 msgr2=0x7fe8441099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.179+0000 7fe84a0b3700 1 --2- 172.21.15.67:0/1197003885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8441095d0 0x7fe8441099b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe8380099c0 tx=0x7fe838009cd0 comp rx=0 tx=0).stop 2024-05-03T17:22:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.179+0000 7fe84a0b3700 1 -- 172.21.15.67:0/1197003885 shutdown_connections 2024-05-03T17:22:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.179+0000 7fe84a0b3700 1 --2- 172.21.15.67:0/1197003885 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844103ef0 0x7fe8441103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.179+0000 7fe84a0b3700 1 --2- 172.21.15.67:0/1197003885 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8441095d0 0x7fe8441099b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.179+0000 7fe84a0b3700 1 --2- 172.21.15.67:0/1197003885 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe844103530 0x7fe8441039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.179+0000 7fe84a0b3700 1 -- 172.21.15.67:0/1197003885 >> 172.21.15.67:0/1197003885 conn(0x7fe84407a690 msgr2=0x7fe84407aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.179+0000 7fe84a0b3700 1 -- 172.21.15.67:0/1197003885 shutdown_connections 2024-05-03T17:22:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.179+0000 7fe84a0b3700 1 -- 172.21.15.67:0/1197003885 wait complete. 2024-05-03T17:22:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe84a0b3700 1 Processor -- start 2024-05-03T17:22:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe84a0b3700 1 -- start start 2024-05-03T17:22:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe84a0b3700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844103530 0x7fe84419f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:57.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe84a0b3700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe844103ef0 0x7fe84419fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe84a0b3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8441095d0 0x7fe844199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe84a0b3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe8441137b0 con 0x7fe8441095d0 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe84a0b3700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe844113630 con 0x7fe844103530 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe84a0b3700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe844113930 con 0x7fe844103ef0 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe843fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8441095d0 0x7fe844199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe843fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8441095d0 0x7fe844199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36730/0 (socket says 172.21.15.67:36730) 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe843fff700 1 -- 172.21.15.67:0/893578460 learned_addr learned my addr 172.21.15.67:0/893578460 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe8437fe700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844103530 0x7fe84419f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.180+0000 7fe842ffd700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe844103ef0 0x7fe84419fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe843fff700 1 -- 172.21.15.67:0/893578460 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe844103ef0 msgr2=0x7fe84419fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe843fff700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe844103ef0 0x7fe84419fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe843fff700 1 -- 172.21.15.67:0/893578460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844103530 msgr2=0x7fe84419f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe843fff700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844103530 0x7fe84419f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe843fff700 1 -- 172.21.15.67:0/893578460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe84419a230 con 0x7fe8441095d0 2024-05-03T17:22:57.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe8437fe700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844103530 0x7fe84419f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:22:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe842ffd700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe844103ef0 0x7fe84419fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:22:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe843fff700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8441095d0 0x7fe844199ad0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00b970 tx=0x7fe83c00bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe840ff9700 1 -- 172.21.15.67:0/893578460 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c019070 con 0x7fe8441095d0 2024-05-03T17:22:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe84a0b3700 1 -- 172.21.15.67:0/893578460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe84419a520 con 0x7fe8441095d0 2024-05-03T17:22:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe840ff9700 1 -- 172.21.15.67:0/893578460 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe83c014440 con 0x7fe8441095d0 2024-05-03T17:22:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe840ff9700 1 -- 172.21.15.67:0/893578460 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c00ce60 con 0x7fe8441095d0 2024-05-03T17:22:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.181+0000 7fe84a0b3700 1 -- 172.21.15.67:0/893578460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8441a67e0 con 0x7fe8441095d0 2024-05-03T17:22:57.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.182+0000 7fe84a0b3700 1 -- 172.21.15.67:0/893578460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe84410ba60 con 0x7fe8441095d0 2024-05-03T17:22:57.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.183+0000 7fe840ff9700 1 -- 172.21.15.67:0/893578460 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fe83c01c070 con 0x7fe8441095d0 2024-05-03T17:22:57.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.184+0000 7fe840ff9700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe82c0b2910 0x7fe82c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:22:57.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.184+0000 7fe8437fe700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe82c0b2910 0x7fe82c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:22:57.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.184+0000 7fe840ff9700 1 -- 172.21.15.67:0/893578460 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fe83c0e1b80 con 0x7fe8441095d0 2024-05-03T17:22:57.186 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.185+0000 7fe8437fe700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe82c0b2910 0x7fe82c0b4dd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe838003800 tx=0x7fe838005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:22:57.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.188+0000 7fe840ff9700 1 -- 172.21.15.67:0/893578460 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe83c0a9840 con 0x7fe8441095d0 2024-05-03T17:22:57.406 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.406+0000 7fe84a0b3700 1 -- 172.21.15.67:0/893578460 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe84419afc0 con 0x7fe8441095d0 2024-05-03T17:22:57.408 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.407+0000 7fe840ff9700 1 -- 172.21.15.67:0/893578460 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22635 (secure 0 0 0) 0x7fe83c0a9140 con 0x7fe8441095d0 2024-05-03T17:22:57.408 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:22:57.408 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":81,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","created":"2024-05-03T17:12:11.709381+0000","modified":"2024-05-03T17:21:07.584731+0000","last_up_change":"2024-05-03T17:19:29.310797+0000","last_in_change":"2024-05-03T17:19:08.848654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T17:15:36.413266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T17:20:31.546843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T17:20:35.261082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80962eb1-a506-49bb-a528-dee035eb4322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6803","nonce":1754365741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6805","nonce":1754365741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6809","nonce":1754365741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6807","nonce":1754365741}]},"public_addr":"172.21.15.67:6803/1754365741","cluster_addr":"172.21.15.67:6805/1754365741","heartbeat_back_addr":"172.21.15.67:6809/1754365741","heartbeat_front_addr":"172.21.15.67:6807/1754365741","state":["exists","up"]},{"osd":1,"uuid":"d7721d5f-4373-4cb2-bb57-bc368d6bb818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6801","nonce":1006339176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6803","nonce":1006339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6807","nonce":1006339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6805","nonce":1006339176}]},"public_addr":"172.21.15.70:6801/1006339176","cluster_addr":"172.21.15.70:6803/1006339176","heartbeat_back_addr":"172.21.15.70:6807/1006339176","heartbeat_front_addr":"172.21.15.70:6805/1006339176","state":["exists","up"]},{"osd":2,"uuid":"9992fd98-821b-40ce-b051-89befbfd87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6801","nonce":1888803264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6803","nonce":1888803264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6807","nonce":1888803264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6805","nonce":1888803264}]},"public_addr":"172.21.15.82:6801/1888803264","cluster_addr":"172.21.15.82:6803/1888803264","heartbeat_back_addr":"172.21.15.82:6807/1888803264","heartbeat_front_addr":"172.21.15.82:6805/1888803264","state":["exists","up"]},{"osd":3,"uuid":"14bfe6a6-0bc8-4766-8d36-fbaecc61a396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6811","nonce":313717723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6813","nonce":313717723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6817","nonce":313717723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6815","nonce":313717723}]},"public_addr":"172.21.15.67:6811/313717723","cluster_addr":"172.21.15.67:6813/313717723","heartbeat_back_addr":"172.21.15.67:6817/313717723","heartbeat_front_addr":"172.21.15.67:6815/313717723","state":["exists","up"]},{"osd":4,"uuid":"5a44aeb8-aabf-4824-be65-2c7170dbee52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6809","nonce":3526300828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6811","nonce":3526300828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6815","nonce":3526300828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6813","nonce":3526300828}]},"public_addr":"172.21.15.70:6809/3526300828","cluster_addr":"172.21.15.70:6811/3526300828","heartbeat_back_addr":"172.21.15.70:6815/3526300828","heartbeat_front_addr":"172.21.15.70:6813/3526300828","state":["exists","up"]},{"osd":5,"uuid":"a6ac56df-466e-4096-8fbf-835f5b544297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6809","nonce":3754223290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6811","nonce":3754223290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6815","nonce":3754223290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6813","nonce":3754223290}]},"public_addr":"172.21.15.82:6809/3754223290","cluster_addr":"172.21.15.82:6811/3754223290","heartbeat_back_addr":"172.21.15.82:6815/3754223290","heartbeat_front_addr":"172.21.15.82:6813/3754223290","state":["exists","up"]},{"osd":6,"uuid":"4d8a5e8a-66ac-462e-b960-24c0ee248fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6819","nonce":4033769161}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6821","nonce":4033769161}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6825","nonce":4033769161}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6823","nonce":4033769161}]},"public_addr":"172.21.15.67:6819/4033769161","cluster_addr":"172.21.15.67:6821/4033769161","heartbeat_back_addr":"172.21.15.67:6825/4033769161","heartbeat_front_addr":"172.21.15.67:6823/4033769161","state":["exists","up"]},{"osd":7,"uuid":"eb983837-8172-4802-9843-6360cda8f4be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6817","nonce":3366146173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6819","nonce":3366146173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6823","nonce":3366146173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6821","nonce":3366146173}]},"public_addr":"172.21.15.70:6817/3366146173","cluster_addr":"172.21.15.70:6819/3366146173","heartbeat_back_addr":"172.21.15.70:6823/3366146173","heartbeat_front_addr":"172.21.15.70:6821/3366146173","state":["exists","up"]},{"osd":8,"uuid":"c9d4ce56-e940-48d1-a9b5-437139e3fed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6817","nonce":347699286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6819","nonce":347699286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6823","nonce":347699286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6821","nonce":347699286}]},"public_addr":"172.21.15.82:6817/347699286","cluster_addr":"172.21.15.82:6819/347699286","heartbeat_back_addr":"172.21.15.82:6823/347699286","heartbeat_front_addr":"172.21.15.82:6821/347699286","state":["exists","up"]},{"osd":9,"uuid":"09596d19-f2d7-485e-a028-9ce5468c6651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6827","nonce":2600095198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6829","nonce":2600095198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6833","nonce":2600095198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6831","nonce":2600095198}]},"public_addr":"172.21.15.67:6827/2600095198","cluster_addr":"172.21.15.67:6829/2600095198","heartbeat_back_addr":"172.21.15.67:6833/2600095198","heartbeat_front_addr":"172.21.15.67:6831/2600095198","state":["exists","up"]},{"osd":10,"uuid":"ec4cffbc-0127-47e5-8176-0972f74a5514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6825","nonce":2897583629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6827","nonce":2897583629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6831","nonce":2897583629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6829","nonce":2897583629}]},"public_addr":"172.21.15.70:6825/2897583629","cluster_addr":"172.21.15.70:6827/2897583629","heartbeat_back_addr":"172.21.15.70:6831/2897583629","heartbeat_front_addr":"172.21.15.70:6829/2897583629","state":["exists","up"]},{"osd":11,"uuid":"a8acfb0b-98e8-49e1-a114-f33aa8160b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6825","nonce":4082656149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6827","nonce":4082656149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6831","nonce":4082656149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6829","nonce":4082656149}]},"public_addr":"172.21.15.82:6825/4082656149","cluster_addr":"172.21.15.82:6827/4082656149","heartbeat_back_addr":"172.21.15.82:6831/4082656149","heartbeat_front_addr":"172.21.15.82:6829/4082656149","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:14:45.596665+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:09.538408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:33.608945+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:59.803102+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:25.187966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:49.892160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:16.575973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:42.115955+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:07.655558+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:35.356779+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:01.144695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:27.577513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/2131289004":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/3335149382":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/992151181":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2614686067":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/3137329076":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2323686292":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/2579128777":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/3335149382":"2024-05-04T17:13:06.324002+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T17:22:57.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.409+0000 7fe84a0b3700 1 -- 172.21.15.67:0/893578460 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe82c0b2910 msgr2=0x7fe82c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:57.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.409+0000 7fe84a0b3700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe82c0b2910 0x7fe82c0b4dd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe838003800 tx=0x7fe838005fb0 comp rx=0 tx=0).stop 2024-05-03T17:22:57.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.409+0000 7fe84a0b3700 1 -- 172.21.15.67:0/893578460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8441095d0 msgr2=0x7fe844199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:22:57.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.409+0000 7fe84a0b3700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8441095d0 0x7fe844199ad0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00b970 tx=0x7fe83c00bd30 comp rx=0 tx=0).stop 2024-05-03T17:22:57.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.409+0000 7fe84a0b3700 1 -- 172.21.15.67:0/893578460 shutdown_connections 2024-05-03T17:22:57.410 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.409+0000 7fe84a0b3700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe82c0b2910 0x7fe82c0b4dd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:57.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.409+0000 7fe84a0b3700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8441095d0 0x7fe844199ad0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:57.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.409+0000 7fe84a0b3700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe844103530 0x7fe84419f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:57.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.409+0000 7fe84a0b3700 1 --2- 172.21.15.67:0/893578460 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe844103ef0 0x7fe84419fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:22:57.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.409+0000 7fe84a0b3700 1 -- 172.21.15.67:0/893578460 >> 172.21.15.67:0/893578460 conn(0x7fe84407a690 msgr2=0x7fe8441022a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:22:57.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.410+0000 7fe84a0b3700 1 -- 172.21.15.67:0/893578460 shutdown_connections 2024-05-03T17:22:57.411 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:22:57.410+0000 7fe84a0b3700 1 -- 172.21.15.67:0/893578460 wait complete. 2024-05-03T17:22:57.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:57 smithi067 ceph-mon[29731]: pgmap v366: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:57.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:57 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/893578460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:22:58.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:57 smithi082 ceph-mon[33535]: pgmap v366: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:58.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:57 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/893578460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:22:58.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:57 smithi070 ceph-mon[33250]: pgmap v366: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:22:58.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:57 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/893578460' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:22:58.165 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:22:58.420 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:22:59.872 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:22:59 smithi067 ceph-mon[29731]: pgmap v367: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:00.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:22:59 smithi082 ceph-mon[33535]: pgmap v367: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:00.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:22:59 smithi070 ceph-mon[33250]: pgmap v367: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.113+0000 7f88385fc700 1 -- 172.21.15.67:0/1244611503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8830109500 msgr2=0x7f88301098e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.113+0000 7f88385fc700 1 --2- 172.21.15.67:0/1244611503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8830109500 0x7f88301098e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f882c009990 tx=0x7f882c009ca0 comp rx=0 tx=0).stop 2024-05-03T17:23:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.113+0000 7f88385fc700 1 -- 172.21.15.67:0/1244611503 shutdown_connections 2024-05-03T17:23:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.113+0000 7f88385fc700 1 --2- 172.21.15.67:0/1244611503 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f883006c3d0 0x7f8830110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.113+0000 7f88385fc700 1 --2- 172.21.15.67:0/1244611503 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8830109500 0x7f88301098e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.113+0000 7f88385fc700 1 --2- 172.21.15.67:0/1244611503 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f883006ba10 0x7f883006be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.113+0000 7f88385fc700 1 -- 172.21.15.67:0/1244611503 >> 172.21.15.67:0/1244611503 conn(0x7f8830079310 msgr2=0x7f8830079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:00.126 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.113+0000 7f88385fc700 1 -- 172.21.15.67:0/1244611503 shutdown_connections 2024-05-03T17:23:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.113+0000 7f88385fc700 1 -- 172.21.15.67:0/1244611503 wait complete. 2024-05-03T17:23:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.114+0000 7f88385fc700 1 Processor -- start 2024-05-03T17:23:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.114+0000 7f88385fc700 1 -- start start 2024-05-03T17:23:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.114+0000 7f88385fc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f883006ba10 0x7f883019f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.114+0000 7f88385fc700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f883006c3d0 0x7f883019fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.114+0000 7f88385fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8830109500 0x7f8830199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.114+0000 7f88385fc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8830113660 con 0x7f8830109500 2024-05-03T17:23:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.114+0000 7f88385fc700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f88301134e0 con 0x7f883006ba10 2024-05-03T17:23:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.114+0000 7f88385fc700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f88301137e0 con 0x7f883006c3d0 2024-05-03T17:23:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f8836b99700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8830109500 0x7f8830199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f8836b99700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8830109500 0x7f8830199980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36756/0 (socket says 172.21.15.67:36756) 2024-05-03T17:23:00.127 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f8836b99700 1 -- 172.21.15.67:0/3326600447 learned_addr learned my addr 172.21.15.67:0/3326600447 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f8836398700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f883006ba10 0x7f883019f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f8835b97700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f883006c3d0 0x7f883019fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f8836b99700 1 -- 172.21.15.67:0/3326600447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f883006c3d0 msgr2=0x7f883019fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f8836b99700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f883006c3d0 0x7f883019fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f8836b99700 1 -- 172.21.15.67:0/3326600447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f883006ba10 msgr2=0x7f883019f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f8836b99700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f883006ba10 0x7f883019f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f8836b99700 1 -- 172.21.15.67:0/3326600447 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f883019a0e0 con 0x7f8830109500 2024-05-03T17:23:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f8836b99700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8830109500 0x7f8830199980 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8828007d70 tx=0x7f882800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f88277fe700 1 -- 172.21.15.67:0/3326600447 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8828016020 con 0x7f8830109500 2024-05-03T17:23:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f88277fe700 1 -- 172.21.15.67:0/3326600447 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f882800ad40 con 0x7f8830109500 2024-05-03T17:23:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f88277fe700 1 -- 172.21.15.67:0/3326600447 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88280115e0 con 0x7f8830109500 2024-05-03T17:23:00.128 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f88385fc700 1 -- 172.21.15.67:0/3326600447 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f883019a3d0 con 0x7f8830109500 2024-05-03T17:23:00.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.115+0000 7f88385fc700 1 -- 172.21.15.67:0/3326600447 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88301a6640 con 0x7f8830109500 2024-05-03T17:23:00.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.116+0000 7f88385fc700 1 -- 172.21.15.67:0/3326600447 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f883010b9e0 con 0x7f8830109500 2024-05-03T17:23:00.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.118+0000 7f88277fe700 1 -- 172.21.15.67:0/3326600447 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f8828019020 con 0x7f8830109500 2024-05-03T17:23:00.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.119+0000 7f88277fe700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f881c0b2910 0x7f881c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:00.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.119+0000 7f8836398700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f881c0b2910 0x7f881c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:00.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.119+0000 7f88277fe700 1 -- 172.21.15.67:0/3326600447 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f88280e2740 con 0x7f8830109500 2024-05-03T17:23:00.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.119+0000 7f8836398700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f881c0b2910 0x7f881c0b4dd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8830100a10 tx=0x7f882c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:00.129 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.122+0000 7f88277fe700 1 -- 172.21.15.67:0/3326600447 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f8828020020 con 0x7f8830109500 2024-05-03T17:23:00.380 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.379+0000 7f88385fc700 1 -- 172.21.15.67:0/3326600447 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f883006a190 con 0x7f8830109500 2024-05-03T17:23:00.381 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.380+0000 7f88277fe700 1 -- 172.21.15.67:0/3326600447 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11850 (secure 0 0 0) 0x7f88280aa400 con 0x7f8830109500 2024-05-03T17:23:00.381 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:23:00.381 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/316463706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":316463706},{"type":"v1","addr":"172.21.15.67:6839","nonce":316463706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3422086111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3422086111},{"type":"v1","addr":"172.21.15.67:6841","nonce":3422086111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T17:20:39.322107+0000","modified":"2024-05-03T17:21:01.753515+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24464,"mds_1":24446,"mds_2":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24421":{"gid":24421,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1568454842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1568454842},{"type":"v1","addr":"172.21.15.82:6835","nonce":1568454842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/370282857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":370282857},{"type":"v1","addr":"172.21.15.82:6837","nonce":370282857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.82:6839/2084218047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2084218047},{"type":"v1","addr":"172.21.15.82:6839","nonce":2084218047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.70:6837/3323713018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3323713018},{"type":"v1","addr":"172.21.15.70:6837","nonce":3323713018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6839/902276547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":902276547},{"type":"v1","addr":"172.21.15.70:6839","nonce":902276547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24446,24464,24442]},"id":1}]} 2024-05-03T17:23:00.383 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.382+0000 7f88385fc700 1 -- 172.21.15.67:0/3326600447 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f881c0b2910 msgr2=0x7f881c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:00.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.382+0000 7f88385fc700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f881c0b2910 0x7f881c0b4dd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8830100a10 tx=0x7f882c00b540 comp rx=0 tx=0).stop 2024-05-03T17:23:00.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.382+0000 7f88385fc700 1 -- 172.21.15.67:0/3326600447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8830109500 msgr2=0x7f8830199980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:00.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.382+0000 7f88385fc700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8830109500 0x7f8830199980 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f8828007d70 tx=0x7f882800a340 comp rx=0 tx=0).stop 2024-05-03T17:23:00.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.382+0000 7f88385fc700 1 -- 172.21.15.67:0/3326600447 shutdown_connections 2024-05-03T17:23:00.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.382+0000 7f88385fc700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f881c0b2910 0x7f881c0b4dd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:00.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.382+0000 7f88385fc700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8830109500 0x7f8830199980 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:00.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.382+0000 7f88385fc700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f883006ba10 0x7f883019f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:00.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.382+0000 7f88385fc700 1 --2- 172.21.15.67:0/3326600447 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f883006c3d0 0x7f883019fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:00.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.382+0000 7f88385fc700 1 -- 172.21.15.67:0/3326600447 >> 172.21.15.67:0/3326600447 conn(0x7f8830079310 msgr2=0x7f8830101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:00.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.382+0000 7f88385fc700 1 -- 172.21.15.67:0/3326600447 shutdown_connections 2024-05-03T17:23:00.384 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:00.382+0000 7f88385fc700 1 -- 172.21.15.67:0/3326600447 wait complete. 2024-05-03T17:23:00.385 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T17:23:00.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:00 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3326600447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:23:01.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:00 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3326600447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:23:01.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:00 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3326600447' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:23:01.111 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-03T17:20:39.322107+0000', 'modified': '2024-05-03T17:21:01.753515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24464, 'mds_1': 24446, 'mds_2': 24442}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24421': {'gid': 24421, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6835/1568454842', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6834', 'nonce': 1568454842}, {'type': 'v1', 'addr': '172.21.15.82:6835', 'nonce': 1568454842}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6835/1966549782', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6834', 'nonce': 1966549782}, {'type': 'v1', 'addr': '172.21.15.70:6835', 'nonce': 1966549782}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24427': {'gid': 24427, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6837/370282857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6836', 'nonce': 370282857}, {'type': 'v1', 'addr': '172.21.15.82:6837', 'nonce': 370282857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.82:6839/2084218047', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6838', 'nonce': 2084218047}, {'type': 'v1', 'addr': '172.21.15.82:6839', 'nonce': 2084218047}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.70:6837/3323713018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 3323713018}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 3323713018}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.70:6839/902276547', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 902276547}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 902276547}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24464, 'qdb_cluster': [24446, 24464, 24442]} 2024-05-03T17:23:01.111 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-03T17:23:01.111 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell mds.i status 2024-05-03T17:23:01.362 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:01.735 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:01 smithi067 ceph-mon[29731]: pgmap v368: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:02.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:01 smithi082 ceph-mon[33535]: pgmap v368: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:02.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:01 smithi070 ceph-mon[33250]: pgmap v368: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.042+0000 7f965e61d700 1 -- 172.21.15.67:0/2014576997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9658103540 msgr2=0x7f96581039c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.042+0000 7f965e61d700 1 --2- 172.21.15.67:0/2014576997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9658103540 0x7f96581039c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f964c0099c0 tx=0x7f964c009cd0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.044 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.042+0000 7f965e61d700 1 -- 172.21.15.67:0/2014576997 shutdown_connections 2024-05-03T17:23:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.042+0000 7f965e61d700 1 --2- 172.21.15.67:0/2014576997 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9658103f00 0x7f96581103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.042+0000 7f965e61d700 1 --2- 172.21.15.67:0/2014576997 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f96581095e0 0x7f96581099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.042+0000 7f965e61d700 1 --2- 172.21.15.67:0/2014576997 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9658103540 0x7f96581039c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.042+0000 7f965e61d700 1 -- 172.21.15.67:0/2014576997 >> 172.21.15.67:0/2014576997 conn(0x7f965807a650 msgr2=0x7f965807aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.042+0000 7f965e61d700 1 -- 172.21.15.67:0/2014576997 shutdown_connections 2024-05-03T17:23:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.042+0000 7f965e61d700 1 -- 172.21.15.67:0/2014576997 wait complete. 2024-05-03T17:23:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.043+0000 7f965e61d700 1 Processor -- start 2024-05-03T17:23:03.045 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.043+0000 7f965e61d700 1 -- start start 2024-05-03T17:23:03.046 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.043+0000 7f965e61d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9658103540 0x7f965819f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.043+0000 7f965e61d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9658103f00 0x7f965819fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.043+0000 7f965e61d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581095e0 0x7f96581999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.043+0000 7f965e61d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9658113760 con 0x7f96581095e0 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.043+0000 7f965e61d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f96581135e0 con 0x7f9658103540 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.043+0000 7f965e61d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f96581138e0 con 0x7f9658103f00 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.043+0000 7f965cbba700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581095e0 0x7f96581999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.043+0000 7f965cbba700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581095e0 0x7f96581999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36778/0 (socket says 172.21.15.67:36778) 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.043+0000 7f965cbba700 1 -- 172.21.15.67:0/3373283765 learned_addr learned my addr 172.21.15.67:0/3373283765 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f96577fe700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9658103f00 0x7f965819fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f9657fff700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9658103540 0x7f965819f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f965cbba700 1 -- 172.21.15.67:0/3373283765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9658103f00 msgr2=0x7f965819fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f965cbba700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9658103f00 0x7f965819fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f965cbba700 1 -- 172.21.15.67:0/3373283765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9658103540 msgr2=0x7f965819f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:03.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f965cbba700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9658103540 0x7f965819f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f965cbba700 1 -- 172.21.15.67:0/3373283765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965819a120 con 0x7f96581095e0 2024-05-03T17:23:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f965cbba700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581095e0 0x7f96581999c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f965000b970 tx=0x7f965000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f96557fa700 1 -- 172.21.15.67:0/3373283765 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965000d860 con 0x7f96581095e0 2024-05-03T17:23:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f965e61d700 1 -- 172.21.15.67:0/3373283765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965819a410 con 0x7f96581095e0 2024-05-03T17:23:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f96557fa700 1 -- 172.21.15.67:0/3373283765 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f965000e430 con 0x7f96581095e0 2024-05-03T17:23:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f96557fa700 1 -- 172.21.15.67:0/3373283765 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650017660 con 0x7f96581095e0 2024-05-03T17:23:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.044+0000 7f965e61d700 1 -- 172.21.15.67:0/3373283765 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96581a66d0 con 0x7f96581095e0 2024-05-03T17:23:03.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.047+0000 7f96557fa700 1 -- 172.21.15.67:0/3373283765 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f965000da00 con 0x7f96581095e0 2024-05-03T17:23:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.047+0000 7f96557fa700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f96400b2910 0x7f96400b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.048+0000 7f9657fff700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f96400b2910 0x7f96400b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.048+0000 7f96557fa700 1 -- 172.21.15.67:0/3373283765 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f96500e1f90 con 0x7f96581095e0 2024-05-03T17:23:03.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.048+0000 7f9657fff700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f96400b2910 0x7f96400b4dd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9648005950 tx=0x7f964800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.048+0000 7f9668eb0b80 1 Processor -- start 2024-05-03T17:23:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.048+0000 7f9668eb0b80 1 -- start start 2024-05-03T17:23:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f9668eb0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ed3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f9668eb0b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 0x55ec49ecef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f9668eb0b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 0x55ec49ecf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f9668eb0b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ec49e218f0 con 0x55ec49ed4060 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f9668eb0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ec49d86d50 con 0x55ec49ed3740 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f9668eb0b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ec49e1d8b0 con 0x55ec49ecf480 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f96577fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 0x55ec49ecef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f96577fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 0x55ec49ecef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36792/0 (socket says 172.21.15.67:36792) 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f96577fe700 1 -- 172.21.15.67:0/2323107638 learned_addr learned my addr 172.21.15.67:0/2323107638 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f9657fff700 1 --2- 172.21.15.67:0/2323107638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ed3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f965cbba700 1 --2- 172.21.15.67:0/2323107638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 0x55ec49ecf8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f96577fe700 1 -- 172.21.15.67:0/2323107638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 msgr2=0x55ec49ecf8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f96577fe700 1 --2- 172.21.15.67:0/2323107638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 0x55ec49ecf8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f96577fe700 1 -- 172.21.15.67:0/2323107638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 msgr2=0x55ec49ed3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f96577fe700 1 --2- 172.21.15.67:0/2323107638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ed3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f96577fe700 1 -- 172.21.15.67:0/2323107638 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec49d6e880 con 0x55ec49ed4060 2024-05-03T17:23:03.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f96577fe700 1 --2- 172.21.15.67:0/2323107638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 0x55ec49ecef40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f964c0099c0 tx=0x7f964c003680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71eac4c99091bbd7 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:03.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.049+0000 7f963e7fc700 1 -- 172.21.15.67:0/2323107638 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c028070 con 0x55ec49ed4060 2024-05-03T17:23:03.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.050+0000 7f963e7fc700 1 -- 172.21.15.67:0/2323107638 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f964c005580 con 0x55ec49ed4060 2024-05-03T17:23:03.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.050+0000 7f963e7fc700 1 -- 172.21.15.67:0/2323107638 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c00bb50 con 0x55ec49ed4060 2024-05-03T17:23:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.050+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2323107638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 msgr2=0x55ec49ecef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.050+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2323107638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 0x55ec49ecef40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f964c0099c0 tx=0x7f964c003680 comp rx=0 tx=0).stop 2024-05-03T17:23:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.050+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2323107638 shutdown_connections 2024-05-03T17:23:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.050+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2323107638 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 0x55ec49ecf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.050+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2323107638 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ed3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.050+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2323107638 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 0x55ec49ecef40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.050+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2323107638 >> 172.21.15.67:0/2323107638 conn(0x55ec49dfbb20 msgr2=0x55ec49ec99a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.050+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2323107638 shutdown_connections 2024-05-03T17:23:03.057 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.050+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2323107638 wait complete. 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.051+0000 7f9668eb0b80 1 -- start start 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.051+0000 7f9668eb0b80 1 Processor -- start 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.051+0000 7f9668eb0b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 0x55ec49edad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.051+0000 7f9668eb0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ed8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.051+0000 7f9668eb0b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 0x55ec49ed86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.051+0000 7f9668eb0b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ec49edbb40 con 0x55ec49ed4060 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.051+0000 7f9668eb0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ec49ddae80 con 0x55ec49ed3740 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.051+0000 7f9668eb0b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ec49edbcc0 con 0x55ec49ecf480 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.051+0000 7f9668eb0b80 10 client.0 _ll_register_callbacks cb 0x55ec49db5560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.051+0000 7f965cbba700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 0x55ec49ed86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.051+0000 7f965cbba700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 0x55ec49ed86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36800/0 (socket says 172.21.15.67:36800) 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.051+0000 7f965cbba700 1 -- 172.21.15.67:0/3707111058 learned_addr learned my addr 172.21.15.67:0/3707111058 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:03.058 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f9657fff700 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 0x55ec49edad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f96577fe700 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ed8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f965cbba700 1 -- 172.21.15.67:0/3707111058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 msgr2=0x55ec49edad80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f965cbba700 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 0x55ec49edad80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f965cbba700 1 -- 172.21.15.67:0/3707111058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 msgr2=0x55ec49ed8190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f965cbba700 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ed8190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f965cbba700 1 -- 172.21.15.67:0/3707111058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec49dd8710 con 0x55ec49ed4060 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f965cbba700 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 0x55ec49ed86d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f96500a93d0 tx=0x7f965000eb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ace33bda43275f server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f96377fe700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f96377fe700 1 -- 172.21.15.67:0/3707111058 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96500ad7f0 con 0x55ec49ed4060 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f96377fe700 1 -- 172.21.15.67:0/3707111058 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f96500a5a30 con 0x55ec49ed4060 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f9668eb0b80 1 -- 172.21.15.67:0/3707111058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f964c005580 con 0x55ec49ed4060 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f96377fe700 1 -- 172.21.15.67:0/3707111058 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f96500b22a0 con 0x55ec49ed4060 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f9668eb0b80 10 client.14931 fetch_fsmap learned FSMap version 25 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f9668eb0b80 1 -- 172.21.15.67:0/3707111058 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ec49eea7d0 con 0x55ec49ed4060 2024-05-03T17:23:03.059 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.052+0000 7f96377fe700 1 -- 172.21.15.67:0/3707111058 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f964c005580 con 0x55ec49ed4060 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.053+0000 7f96377fe700 1 -- 172.21.15.67:0/3707111058 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f965001f030 con 0x55ec49ed4060 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.053+0000 7f9668eb0b80 10 client.14931 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.053+0000 7f9668eb0b80 10 client.14931 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.053+0000 7f9668eb0b80 20 client.14931 populate_metadata read hostname 'smithi067' 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.053+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55ec49eecd40 0x55ec49eee1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.053+0000 7f9668eb0b80 4 client.14931 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.053+0000 7f9668eb0b80 1 -- 172.21.15.67:0/3707111058 --> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec49e2f2d0 con 0x55ec49eecd40 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.053+0000 7f9657fff700 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55ec49eecd40 0x55ec49eee1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.053+0000 7f9657fff700 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55ec49eecd40 0x55ec49eee1b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a695c6cd4c091652 server_cookie=f79dd52557b82e0 in_seq=0 out_seq=0 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.053+0000 7f96377fe700 10 client.14931 ms_handle_connect on v2:172.21.15.82:6838/2084218047 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.054+0000 7f96377fe700 1 -- 172.21.15.67:0/3707111058 <== mds.2 v2:172.21.15.82:6838/2084218047 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ec49e2f2d0 con 0x55ec49eecd40 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.054+0000 7f96377fe700 10 client.14931 handle_command_reply: tid=0 2024-05-03T17:23:03.060 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.054+0000 7f9668eb0b80 1 client.14931 shutdown 2024-05-03T17:23:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.054+0000 7f9668eb0b80 1 -- 172.21.15.67:0/3707111058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 msgr2=0x55ec49ed86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.054+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 0x55ec49ed86d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f96500a93d0 tx=0x7f965000eb10 comp rx=0 tx=0).stop 2024-05-03T17:23:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.054+0000 7f9668eb0b80 1 -- 172.21.15.67:0/3707111058 shutdown_connections 2024-05-03T17:23:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.054+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55ec49eecd40 0x55ec49eee1b0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.054+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed4060 0x55ec49ed86d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.054+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 0x55ec49edad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.054+0000 7f96377fe700 0 client.14931 ms_handle_reset on v2:172.21.15.82:6838/2084218047 2024-05-03T17:23:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.054+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/3707111058 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ed8190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.054+0000 7f9668eb0b80 1 -- 172.21.15.67:0/3707111058 >> 172.21.15.67:0/3707111058 conn(0x55ec49dfbb20 msgr2=0x55ec49ed4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.055+0000 7f9668eb0b80 1 -- 172.21.15.67:0/3707111058 shutdown_connections 2024-05-03T17:23:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.055+0000 7f9668eb0b80 1 -- 172.21.15.67:0/3707111058 wait complete. 2024-05-03T17:23:03.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.055+0000 7f9668eb0b80 20 client.14931 trim_cache size 0 max 16384 2024-05-03T17:23:03.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f9668eb0b80 1 Processor -- start 2024-05-03T17:23:03.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f9668eb0b80 1 -- start start 2024-05-03T17:23:03.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f9668eb0b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 0x55ec49ec9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f9668eb0b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed3740 0x55ec49eccf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f9668eb0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed4060 0x55ec49ecd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f9668eb0b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ec49eec4f0 con 0x55ec49ed3740 2024-05-03T17:23:03.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f9668eb0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ec49eecbc0 con 0x55ec49ed4060 2024-05-03T17:23:03.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f9668eb0b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ec49eed290 con 0x55ec49ecf480 2024-05-03T17:23:03.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f96577fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed3740 0x55ec49eccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f96577fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed3740 0x55ec49eccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36810/0 (socket says 172.21.15.67:36810) 2024-05-03T17:23:03.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f96577fe700 1 -- 172.21.15.67:0/2370012900 learned_addr learned my addr 172.21.15.67:0/2370012900 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:03.082 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f9657fff700 1 --2- 172.21.15.67:0/2370012900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 0x55ec49ec9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f96577fe700 1 -- 172.21.15.67:0/2370012900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 msgr2=0x55ec49ec9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f96577fe700 1 --2- 172.21.15.67:0/2370012900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 0x55ec49ec9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f96577fe700 1 -- 172.21.15.67:0/2370012900 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed4060 msgr2=0x55ec49ecd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f96577fe700 1 --2- 172.21.15.67:0/2370012900 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed4060 0x55ec49ecd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.076+0000 7f96577fe700 1 -- 172.21.15.67:0/2370012900 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec49ed3390 con 0x55ec49ed3740 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f96577fe700 1 --2- 172.21.15.67:0/2370012900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed3740 0x55ec49eccf30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f964c009890 tx=0x7f964c01a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30b8ec22b64a9ebf server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f963dffb700 1 -- 172.21.15.67:0/2370012900 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c028040 con 0x55ec49ed3740 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f963dffb700 1 -- 172.21.15.67:0/2370012900 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f964c0059a0 con 0x55ec49ed3740 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f963dffb700 1 -- 172.21.15.67:0/2370012900 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c00bb90 con 0x55ec49ed3740 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2370012900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed3740 msgr2=0x55ec49eccf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2370012900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed3740 0x55ec49eccf30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f964c009890 tx=0x7f964c01a040 comp rx=0 tx=0).stop 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2370012900 shutdown_connections 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2370012900 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed4060 0x55ec49ecd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.083 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2370012900 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ecf480 0x55ec49ec9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2370012900 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ed3740 0x55ec49eccf30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2370012900 >> 172.21.15.67:0/2370012900 conn(0x55ec49dfbb20 msgr2=0x55ec49ed6ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2370012900 shutdown_connections 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.077+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2370012900 wait complete. 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f9668eb0b80 1 -- start start 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f9668eb0b80 1 Processor -- start 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f9668eb0b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ecf480 0x55ec49eccc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f9668eb0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ecd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f9668eb0b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ed4060 0x55ec49ef3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f9668eb0b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55ec49eee030 con 0x55ec49ecf480 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f9668eb0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55ec49eed960 con 0x55ec49ed3740 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f9668eb0b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55ec49eee700 con 0x55ec49ed4060 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f9668eb0b80 10 client.0 _ll_register_callbacks cb 0x55ec49db5560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f9657fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ecf480 0x55ec49eccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f9657fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ecf480 0x55ec49eccc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36818/0 (socket says 172.21.15.67:36818) 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f9657fff700 1 -- 172.21.15.67:0/2225934018 learned_addr learned my addr 172.21.15.67:0/2225934018 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:03.084 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.078+0000 7f965cbba700 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ed4060 0x55ec49ef3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f96577fe700 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ecd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f9657fff700 1 -- 172.21.15.67:0/2225934018 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ed4060 msgr2=0x55ec49ef3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f9657fff700 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ed4060 0x55ec49ef3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f9657fff700 1 -- 172.21.15.67:0/2225934018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 msgr2=0x55ec49ecd1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f9657fff700 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ecd1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f9657fff700 1 -- 172.21.15.67:0/2225934018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec49ef4180 con 0x55ec49ecf480 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f96577fe700 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ecd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f9657fff700 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ecf480 0x55ec49eccc80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55ec49ed8c90 tx=0x7f96480047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383f872044836de4 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f96377fe700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f96377fe700 1 -- 172.21.15.67:0/2225934018 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964802c070 con 0x55ec49ecf480 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f96377fe700 1 -- 172.21.15.67:0/2225934018 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f9648028790 con 0x55ec49ecf480 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2225934018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f964c0059a0 con 0x55ec49ecf480 2024-05-03T17:23:03.085 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f96377fe700 1 -- 172.21.15.67:0/2225934018 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f9648016070 con 0x55ec49ecf480 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f9668eb0b80 10 client.14937 fetch_fsmap learned FSMap version 25 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2225934018 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ec49f58880 con 0x55ec49ecf480 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.079+0000 7f96377fe700 1 -- 172.21.15.67:0/2225934018 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f964c0059a0 con 0x55ec49ecf480 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.080+0000 7f96377fe700 1 -- 172.21.15.67:0/2225934018 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f9648031cd0 con 0x55ec49ecf480 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.080+0000 7f9668eb0b80 10 client.14937 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.080+0000 7f9668eb0b80 10 client.14937 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.080+0000 7f9668eb0b80 20 client.14937 populate_metadata read hostname 'smithi067' 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.080+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55ec49f5d500 0x55ec49f5c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.080+0000 7f9668eb0b80 4 client.14937 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.080+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2225934018 --> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] -- command(tid 0: {"prefix": "status"}) -- 0x55ec49e29370 con 0x55ec49f5d500 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.080+0000 7f96577fe700 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55ec49f5d500 0x55ec49f5c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.080+0000 7f96377fe700 10 client.14937 ms_handle_connect on v2:172.21.15.82:6838/2084218047 2024-05-03T17:23:03.086 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.080+0000 7f96577fe700 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55ec49f5d500 0x55ec49f5c210 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cd991f58c950bc76 server_cookie=a106fa749e30b58a in_seq=0 out_seq=0 2024-05-03T17:23:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f96377fe700 1 -- 172.21.15.67:0/2225934018 <== mds.2 v2:172.21.15.82:6838/2084218047 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ec49e29370 con 0x55ec49f5d500 2024-05-03T17:23:03.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f96377fe700 10 client.14937 handle_command_reply: tid=0 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f9668eb0b80 1 client.14937 shutdown 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2225934018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ecf480 msgr2=0x55ec49eccc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ecf480 0x55ec49eccc80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x55ec49ed8c90 tx=0x7f96480047c0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2225934018 shutdown_connections 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55ec49f5d500 0x55ec49f5c210 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55ec49ed4060 0x55ec49ef3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f96377fe700 0 client.14937 ms_handle_reset on v2:172.21.15.82:6838/2084218047 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55ec49ecf480 0x55ec49eccc80 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f9668eb0b80 1 --2- 172.21.15.67:0/2225934018 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55ec49ed3740 0x55ec49ecd1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2225934018 >> 172.21.15.67:0/2225934018 conn(0x55ec49dfbb20 msgr2=0x55ec49ed2c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2225934018 shutdown_connections 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f9668eb0b80 1 -- 172.21.15.67:0/2225934018 wait complete. 2024-05-03T17:23:03.088 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.081+0000 7f9668eb0b80 20 client.14937 trim_cache size 0 max 16384 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "20c8e22e-0970-11ef-bc97-c7b262605968", 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 2, 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24442, 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 127.53134324299999, 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 81, 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 79, 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 170.48239175800001 2024-05-03T17:23:03.089 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T17:23:03.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.082+0000 7f965e61d700 1 -- 172.21.15.67:0/3373283765 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f96400b2910 msgr2=0x7f96400b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:03.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.082+0000 7f965e61d700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f96400b2910 0x7f96400b4dd0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9648005950 tx=0x7f964800a300 comp rx=0 tx=0).stop 2024-05-03T17:23:03.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.082+0000 7f965e61d700 1 -- 172.21.15.67:0/3373283765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581095e0 msgr2=0x7f96581999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:03.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.082+0000 7f965e61d700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581095e0 0x7f96581999c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f965000b970 tx=0x7f965000bc80 comp rx=0 tx=0).stop 2024-05-03T17:23:03.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.082+0000 7f965e61d700 1 -- 172.21.15.67:0/3373283765 shutdown_connections 2024-05-03T17:23:03.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.082+0000 7f965e61d700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f96400b2910 0x7f96400b4dd0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.082+0000 7f965e61d700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f96581095e0 0x7f96581999c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.082+0000 7f965e61d700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9658103540 0x7f965819f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.082+0000 7f965e61d700 1 --2- 172.21.15.67:0/3373283765 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9658103f00 0x7f965819fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.082+0000 7f965e61d700 1 -- 172.21.15.67:0/3373283765 >> 172.21.15.67:0/3373283765 conn(0x7f965807a650 msgr2=0x7f9658102260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.082+0000 7f965e61d700 1 -- 172.21.15.67:0/3373283765 shutdown_connections 2024-05-03T17:23:03.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:03.082+0000 7f965e61d700 1 -- 172.21.15.67:0/3373283765 wait complete. 2024-05-03T17:23:03.855 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell mds.h status 2024-05-03T17:23:03.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:03 smithi067 ceph-mon[29731]: pgmap v369: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:04.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:03 smithi082 ceph-mon[33535]: pgmap v369: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:04.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:03 smithi070 ceph-mon[33250]: pgmap v369: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:04.105 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:05.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.825+0000 7f4a80a0d700 1 -- 172.21.15.67:0/417662032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c06c630 msgr2=0x7f4a7c110360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:05.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.825+0000 7f4a80a0d700 1 --2- 172.21.15.67:0/417662032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c06c630 0x7f4a7c110360 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4a7400b0d0 tx=0x7f4a7400b3e0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.825+0000 7f4a80a0d700 1 -- 172.21.15.67:0/417662032 shutdown_connections 2024-05-03T17:23:05.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.825+0000 7f4a80a0d700 1 --2- 172.21.15.67:0/417662032 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c06c630 0x7f4a7c110360 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.825+0000 7f4a80a0d700 1 --2- 172.21.15.67:0/417662032 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a7c109590 0x7f4a7c109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.825+0000 7f4a80a0d700 1 --2- 172.21.15.67:0/417662032 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a7c06bc70 0x7f4a7c06c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.825+0000 7f4a80a0d700 1 -- 172.21.15.67:0/417662032 >> 172.21.15.67:0/417662032 conn(0x7f4a7c079440 msgr2=0x7f4a7c079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:05.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.825+0000 7f4a80a0d700 1 -- 172.21.15.67:0/417662032 shutdown_connections 2024-05-03T17:23:05.827 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.825+0000 7f4a80a0d700 1 -- 172.21.15.67:0/417662032 wait complete. 2024-05-03T17:23:05.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.826+0000 7f4a80a0d700 1 Processor -- start 2024-05-03T17:23:05.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.826+0000 7f4a80a0d700 1 -- start start 2024-05-03T17:23:05.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.826+0000 7f4a80a0d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a7c06bc70 0x7f4a7c19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.829 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.826+0000 7f4a80a0d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a7c06c630 0x7f4a7c19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.826+0000 7f4a80a0d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c109590 0x7f4a7c199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.826+0000 7f4a80a0d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7c1136b0 con 0x7f4a7c109590 2024-05-03T17:23:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.826+0000 7f4a80a0d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7c113530 con 0x7f4a7c06bc70 2024-05-03T17:23:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.826+0000 7f4a80a0d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4a7c113830 con 0x7f4a7c06c630 2024-05-03T17:23:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.826+0000 7f4a7ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c109590 0x7f4a7c199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.826+0000 7f4a7ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c109590 0x7f4a7c199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36842/0 (socket says 172.21.15.67:36842) 2024-05-03T17:23:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.826+0000 7f4a7ad9d700 1 -- 172.21.15.67:0/2925498316 learned_addr learned my addr 172.21.15.67:0/2925498316 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a79d9b700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a7c06c630 0x7f4a7c19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a7c06bc70 0x7f4a7c19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a7ad9d700 1 -- 172.21.15.67:0/2925498316 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a7c06c630 msgr2=0x7f4a7c19fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a7c06c630 0x7f4a7c19fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.830 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a7ad9d700 1 -- 172.21.15.67:0/2925498316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a7c06bc70 msgr2=0x7f4a7c19f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:05.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a7c06bc70 0x7f4a7c19f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a7ad9d700 1 -- 172.21.15.67:0/2925498316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a7c199ff0 con 0x7f4a7c109590 2024-05-03T17:23:05.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c109590 0x7f4a7c199890 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4a740095c0 tx=0x7f4a74007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:05.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a737fe700 1 -- 172.21.15.67:0/2925498316 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7401f070 con 0x7f4a7c109590 2024-05-03T17:23:05.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a80a0d700 1 -- 172.21.15.67:0/2925498316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a7c19a280 con 0x7f4a7c109590 2024-05-03T17:23:05.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a737fe700 1 -- 172.21.15.67:0/2925498316 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a74007e20 con 0x7f4a7c109590 2024-05-03T17:23:05.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a737fe700 1 -- 172.21.15.67:0/2925498316 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7401b4a0 con 0x7f4a7c109590 2024-05-03T17:23:05.831 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.827+0000 7f4a80a0d700 1 -- 172.21.15.67:0/2925498316 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a7c1a65a0 con 0x7f4a7c109590 2024-05-03T17:23:05.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.830+0000 7f4a737fe700 1 -- 172.21.15.67:0/2925498316 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4a74022070 con 0x7f4a7c109590 2024-05-03T17:23:05.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.830+0000 7f4a737fe700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4a680b2910 0x7f4a680b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.830+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4a680b2910 0x7f4a680b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a737fe700 1 -- 172.21.15.67:0/2925498316 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f4a7400f020 con 0x7f4a7c109590 2024-05-03T17:23:05.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a8b2a0b80 1 Processor -- start 2024-05-03T17:23:05.832 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4a680b2910 0x7f4a680b4dd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c19a9d0 tx=0x7f4a6400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:05.833 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a8b2a0b80 1 -- start start 2024-05-03T17:23:05.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a8b2a0b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730020 0x557440730400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a8b2a0b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440730940 0x557440734100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a8b2a0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440734640 0x557440734aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a8b2a0b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55744067e810 con 0x557440730020 2024-05-03T17:23:05.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a8b2a0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55744061d770 con 0x557440734640 2024-05-03T17:23:05.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a8b2a0b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55744067a7d0 con 0x557440730940 2024-05-03T17:23:05.837 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a7a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730020 0x557440730400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a7a59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730020 0x557440730400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36850/0 (socket says 172.21.15.67:36850) 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.831+0000 7f4a7a59c700 1 -- 172.21.15.67:0/2022094865 learned_addr learned my addr 172.21.15.67:0/2022094865 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.832+0000 7f4a79d9b700 1 --2- 172.21.15.67:0/2022094865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440730940 0x557440734100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.832+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/2022094865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440734640 0x557440734aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.832+0000 7f4a7a59c700 1 -- 172.21.15.67:0/2022094865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440730940 msgr2=0x557440734100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.832+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/2022094865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440730940 0x557440734100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.832+0000 7f4a7a59c700 1 -- 172.21.15.67:0/2022094865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440734640 msgr2=0x557440734aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.832+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/2022094865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440734640 0x557440734aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.832+0000 7f4a7a59c700 1 -- 172.21.15.67:0/2022094865 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574405cb880 con 0x557440730020 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.832+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/2022094865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440734640 0x557440734aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.832+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/2022094865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730020 0x557440730400 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4a64010f70 tx=0x7f4a64013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7e69ed80c29bdd server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.832+0000 7f4a70ff9700 1 -- 172.21.15.67:0/2022094865 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6400be60 con 0x557440730020 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.832+0000 7f4a70ff9700 1 -- 172.21.15.67:0/2022094865 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a6401ead0 con 0x557440730020 2024-05-03T17:23:05.838 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.832+0000 7f4a70ff9700 1 -- 172.21.15.67:0/2022094865 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6401cae0 con 0x557440730020 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.833+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/2022094865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730020 msgr2=0x557440730400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.833+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/2022094865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730020 0x557440730400 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f4a64010f70 tx=0x7f4a64013000 comp rx=0 tx=0).stop 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.833+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/2022094865 shutdown_connections 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.833+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/2022094865 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440734640 0x557440734aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.833+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/2022094865 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730020 0x557440730400 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.833+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/2022094865 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440730940 0x557440734100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.833+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/2022094865 >> 172.21.15.67:0/2022094865 conn(0x5574406580b0 msgr2=0x557440726280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.833+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/2022094865 shutdown_connections 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.833+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/2022094865 wait complete. 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.833+0000 7f4a8b2a0b80 1 -- start start 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.833+0000 7f4a8b2a0b80 1 Processor -- start 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a8b2a0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730020 0x55744073ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a8b2a0b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 0x557440738db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a8b2a0b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440734640 0x5574407392f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a8b2a0b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55744073c8a0 con 0x557440730940 2024-05-03T17:23:05.839 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a8b2a0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5574405e4be0 con 0x557440730020 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a8b2a0b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55744073ca20 con 0x557440734640 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a8b2a0b80 10 client.0 _ll_register_callbacks cb 0x5574406168d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a79d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 0x557440738db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a79d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 0x557440738db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36860/0 (socket says 172.21.15.67:36860) 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a79d9b700 1 -- 172.21.15.67:0/3673123152 learned_addr learned my addr 172.21.15.67:0/3673123152 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730020 0x55744073ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440734640 0x5574407392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a79d9b700 1 -- 172.21.15.67:0/3673123152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440734640 msgr2=0x5574407392f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a79d9b700 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440734640 0x5574407392f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a79d9b700 1 -- 172.21.15.67:0/3673123152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730020 msgr2=0x55744073ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a79d9b700 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730020 0x55744073ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a79d9b700 1 -- 172.21.15.67:0/3673123152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55744061a640 con 0x557440730940 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a79d9b700 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 0x557440738db0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00b960 tx=0x7f4a6c00bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6503ae10167048 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:05.840 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a61ffb700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T17:23:05.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.834+0000 7f4a61ffb700 1 -- 172.21.15.67:0/3673123152 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a6c01b440 con 0x557440730940 2024-05-03T17:23:05.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a61ffb700 1 -- 172.21.15.67:0/3673123152 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f4a6c01b5e0 con 0x557440730940 2024-05-03T17:23:05.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/3673123152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a6401ead0 con 0x557440730940 2024-05-03T17:23:05.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a61ffb700 1 -- 172.21.15.67:0/3673123152 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4a6c01b7e0 con 0x557440730940 2024-05-03T17:23:05.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a8b2a0b80 10 client.14949 fetch_fsmap learned FSMap version 25 2024-05-03T17:23:05.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/3673123152 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5574407470d0 con 0x557440730940 2024-05-03T17:23:05.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a61ffb700 1 -- 172.21.15.67:0/3673123152 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a6401ead0 con 0x557440730940 2024-05-03T17:23:05.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a61ffb700 1 -- 172.21.15.67:0/3673123152 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f4a6c019020 con 0x557440730940 2024-05-03T17:23:05.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a8b2a0b80 10 client.14949 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T17:23:05.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a8b2a0b80 10 client.14949 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-03T17:23:05.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a8b2a0b80 20 client.14949 populate_metadata read hostname 'smithi067' 2024-05-03T17:23:05.841 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x557440749770 0x55744074abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a8b2a0b80 4 client.14949 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-03T17:23:05.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/3673123152 --> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55744068c1f0 con 0x557440749770 2024-05-03T17:23:05.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.835+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x557440749770 0x55744074abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.836+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x557440749770 0x55744074abe0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dcce512d673132a2 server_cookie=90fb47f47e774a6f in_seq=0 out_seq=0 2024-05-03T17:23:05.842 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.836+0000 7f4a61ffb700 10 client.14949 ms_handle_connect on v2:172.21.15.70:6836/3323713018 2024-05-03T17:23:05.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a61ffb700 1 -- 172.21.15.67:0/3673123152 <== mds.1 v2:172.21.15.70:6836/3323713018 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55744068c1f0 con 0x557440749770 2024-05-03T17:23:05.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a61ffb700 10 client.14949 handle_command_reply: tid=0 2024-05-03T17:23:05.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a8b2a0b80 1 client.14949 shutdown 2024-05-03T17:23:05.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/3673123152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 msgr2=0x557440738db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:05.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 0x557440738db0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4a6c00b960 tx=0x7f4a6c00bd20 comp rx=0 tx=0).stop 2024-05-03T17:23:05.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/3673123152 shutdown_connections 2024-05-03T17:23:05.843 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440734640 0x5574407392f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730020 0x55744073ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x557440749770 0x55744074abe0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/3673123152 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 0x557440738db0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/3673123152 >> 172.21.15.67:0/3673123152 conn(0x5574406580b0 msgr2=0x557440735af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:05.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a61ffb700 0 client.14949 ms_handle_reset on v2:172.21.15.70:6836/3323713018 2024-05-03T17:23:05.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/3673123152 shutdown_connections 2024-05-03T17:23:05.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/3673123152 wait complete. 2024-05-03T17:23:05.844 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.837+0000 7f4a8b2a0b80 20 client.14949 trim_cache size 0 max 16384 2024-05-03T17:23:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a8b2a0b80 1 Processor -- start 2024-05-03T17:23:05.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a8b2a0b80 1 -- start start 2024-05-03T17:23:05.860 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:05 smithi067 ceph-mon[29731]: pgmap v370: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a8b2a0b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440730020 0x557440727d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a8b2a0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730940 0x557440737d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a8b2a0b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440734640 0x55744072b130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a8b2a0b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x557440748f20 con 0x557440734640 2024-05-03T17:23:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a8b2a0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x5574407495f0 con 0x557440730940 2024-05-03T17:23:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a8b2a0b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x557440749cc0 con 0x557440730020 2024-05-03T17:23:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a7ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440734640 0x55744072b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a7ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440734640 0x55744072b130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36868/0 (socket says 172.21.15.67:36868) 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a7ad9d700 1 -- 172.21.15.67:0/1421058959 learned_addr learned my addr 172.21.15.67:0/1421058959 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/1421058959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440730020 0x557440727d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a79d9b700 1 --2- 172.21.15.67:0/1421058959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730940 0x557440737d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a7ad9d700 1 -- 172.21.15.67:0/1421058959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440730020 msgr2=0x557440727d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/1421058959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440730020 0x557440727d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a7ad9d700 1 -- 172.21.15.67:0/1421058959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730940 msgr2=0x557440737d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/1421058959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730940 0x557440737d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.859+0000 7f4a7ad9d700 1 -- 172.21.15.67:0/1421058959 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5574407211f0 con 0x557440734640 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.860+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/1421058959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440734640 0x55744072b130 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009fd0 tx=0x7f4a74092040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7463716007c4a561 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.860+0000 7f4a63fff700 1 -- 172.21.15.67:0/1421058959 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a740b3040 con 0x557440734640 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.860+0000 7f4a63fff700 1 -- 172.21.15.67:0/1421058959 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a740aa5e0 con 0x557440734640 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.860+0000 7f4a63fff700 1 -- 172.21.15.67:0/1421058959 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a740074e0 con 0x557440734640 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.861+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/1421058959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440734640 msgr2=0x55744072b130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:05.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.861+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/1421058959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440734640 0x55744072b130 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f4a74009fd0 tx=0x7f4a74092040 comp rx=0 tx=0).stop 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.861+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/1421058959 shutdown_connections 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.861+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/1421058959 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440734640 0x55744072b130 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.861+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/1421058959 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440730020 0x557440727d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.861+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/1421058959 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730940 0x557440737d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.861+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/1421058959 >> 172.21.15.67:0/1421058959 conn(0x5574406580b0 msgr2=0x55744072aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.861+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/1421058959 shutdown_connections 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.861+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/1421058959 wait complete. 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.861+0000 7f4a8b2a0b80 1 -- start start 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.861+0000 7f4a8b2a0b80 1 Processor -- start 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a8b2a0b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730020 0x557440731b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a8b2a0b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 0x5574407320c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a8b2a0b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440734640 0x557440732600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a8b2a0b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55744074aa60 con 0x557440730940 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a8b2a0b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55744074a390 con 0x557440730020 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a8b2a0b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55744074b130 con 0x557440734640 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a8b2a0b80 10 client.0 _ll_register_callbacks cb 0x5574406168d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a79d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 0x5574407320c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a79d9b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 0x5574407320c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:36880/0 (socket says 172.21.15.67:36880) 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a79d9b700 1 -- 172.21.15.67:0/37109148 learned_addr learned my addr 172.21.15.67:0/37109148 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440734640 0x557440732600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730020 0x557440731b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a7ad9d700 1 -- 172.21.15.67:0/37109148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730020 msgr2=0x557440731b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730020 0x557440731b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a7ad9d700 1 -- 172.21.15.67:0/37109148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 msgr2=0x5574407320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 0x5574407320c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a7ad9d700 1 -- 172.21.15.67:0/37109148 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557440752610 con 0x557440734640 2024-05-03T17:23:05.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730020 0x557440731b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:23:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a7ad9d700 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440734640 0x557440732600 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x557440746fd0 tx=0x7f4a74009e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24097259403f40fd server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a61ffb700 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-05-03T17:23:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a61ffb700 1 -- 172.21.15.67:0/37109148 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a740c3070 con 0x557440734640 2024-05-03T17:23:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a61ffb700 1 -- 172.21.15.67:0/37109148 <== mon.1 v2:172.21.15.82:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f4a740a70c0 con 0x557440734640 2024-05-03T17:23:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.862+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/37109148 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a740aa5e0 con 0x557440734640 2024-05-03T17:23:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.863+0000 7f4a61ffb700 1 -- 172.21.15.67:0/37109148 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4a740b3040 con 0x557440734640 2024-05-03T17:23:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.863+0000 7f4a8b2a0b80 10 client.24772 fetch_fsmap learned FSMap version 25 2024-05-03T17:23:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.863+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/37109148 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5574407b5060 con 0x557440734640 2024-05-03T17:23:05.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.863+0000 7f4a61ffb700 1 -- 172.21.15.67:0/37109148 <== mon.1 v2:172.21.15.82:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a740aa5e0 con 0x557440734640 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.863+0000 7f4a61ffb700 1 -- 172.21.15.67:0/37109148 <== mon.1 v2:172.21.15.82:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f4a74011070 con 0x557440734640 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.863+0000 7f4a8b2a0b80 10 client.24772 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.863+0000 7f4a8b2a0b80 10 client.24772 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.863+0000 7f4a8b2a0b80 20 client.24772 populate_metadata read hostname 'smithi067' 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.863+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x5574407b9a30 0x5574407b8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.863+0000 7f4a8b2a0b80 4 client.24772 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.863+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/37109148 --> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] -- command(tid 0: {"prefix": "status"}) -- 0x557440686290 con 0x5574407b9a30 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.863+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x5574407b9a30 0x5574407b8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a7a59c700 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x5574407b9a30 0x5574407b8740 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2b61d7d81bbc9c3c server_cookie=d92f223466242ff1 in_seq=0 out_seq=0 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a61ffb700 10 client.24772 ms_handle_connect on v2:172.21.15.70:6836/3323713018 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a61ffb700 1 -- 172.21.15.67:0/37109148 <== mds.1 v2:172.21.15.70:6836/3323713018 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557440686290 con 0x5574407b9a30 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a61ffb700 10 client.24772 handle_command_reply: tid=0 2024-05-03T17:23:05.870 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a8b2a0b80 1 client.24772 shutdown 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/37109148 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440734640 msgr2=0x557440732600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440734640 0x557440732600 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x557440746fd0 tx=0x7f4a74009e20 comp rx=0 tx=0).stop 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/37109148 shutdown_connections 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x557440734640 0x557440732600 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x557440730020 0x557440731b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x5574407b9a30 0x5574407b8740 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a8b2a0b80 1 --2- 172.21.15.67:0/37109148 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x557440730940 0x5574407320c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a61ffb700 0 client.24772 ms_handle_reset on v2:172.21.15.70:6836/3323713018 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/37109148 >> 172.21.15.67:0/37109148 conn(0x5574406580b0 msgr2=0x55744072aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/37109148 shutdown_connections 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.864+0000 7f4a8b2a0b80 1 -- 172.21.15.67:0/37109148 wait complete. 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.865+0000 7f4a8b2a0b80 20 client.24772 trim_cache size 0 max 16384 2024-05-03T17:23:05.871 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T17:23:05.872 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "20c8e22e-0970-11ef-bc97-c7b262605968", 2024-05-03T17:23:05.872 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 1, 2024-05-03T17:23:05.872 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24446, 2024-05-03T17:23:05.872 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-03T17:23:05.872 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-03T17:23:05.872 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-03T17:23:05.872 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 131.323743987, 2024-05-03T17:23:05.872 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-03T17:23:05.872 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 81, 2024-05-03T17:23:05.872 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 79, 2024-05-03T17:23:05.872 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 176.17478824200001 2024-05-03T17:23:05.872 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T17:23:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.865+0000 7f4a80a0d700 1 -- 172.21.15.67:0/2925498316 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4a680b2910 msgr2=0x7f4a680b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.865+0000 7f4a80a0d700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4a680b2910 0x7f4a680b4dd0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c19a9d0 tx=0x7f4a6400b410 comp rx=0 tx=0).stop 2024-05-03T17:23:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.865+0000 7f4a80a0d700 1 -- 172.21.15.67:0/2925498316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c109590 msgr2=0x7f4a7c199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.865+0000 7f4a80a0d700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c109590 0x7f4a7c199890 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f4a740095c0 tx=0x7f4a74007a20 comp rx=0 tx=0).stop 2024-05-03T17:23:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.865+0000 7f4a80a0d700 1 -- 172.21.15.67:0/2925498316 shutdown_connections 2024-05-03T17:23:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.865+0000 7f4a80a0d700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4a680b2910 0x7f4a680b4dd0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.865+0000 7f4a80a0d700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4a7c109590 0x7f4a7c199890 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.865+0000 7f4a80a0d700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4a7c06bc70 0x7f4a7c19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.865+0000 7f4a80a0d700 1 --2- 172.21.15.67:0/2925498316 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4a7c06c630 0x7f4a7c19fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.865+0000 7f4a80a0d700 1 -- 172.21.15.67:0/2925498316 >> 172.21.15.67:0/2925498316 conn(0x7f4a7c079440 msgr2=0x7f4a7c100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:05.873 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.866+0000 7f4a80a0d700 1 -- 172.21.15.67:0/2925498316 shutdown_connections 2024-05-03T17:23:05.874 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:05.866+0000 7f4a80a0d700 1 -- 172.21.15.67:0/2925498316 wait complete. 2024-05-03T17:23:06.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:05 smithi082 ceph-mon[33535]: pgmap v370: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:06.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:05 smithi070 ceph-mon[33250]: pgmap v370: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:06.619 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell mds.b status 2024-05-03T17:23:06.876 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:07.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:07 smithi067 ceph-mon[29731]: pgmap v371: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:08.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:07 smithi082 ceph-mon[33535]: pgmap v371: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:08.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:07 smithi070 ceph-mon[33250]: pgmap v371: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:08.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.661+0000 7f078e182700 1 -- 172.21.15.67:0/1339027081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0788075ff0 msgr2=0x7f078810f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:08.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.661+0000 7f078e182700 1 --2- 172.21.15.67:0/1339027081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0788075ff0 0x7f078810f990 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f077c009990 tx=0x7f077c009ca0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.661+0000 7f078e182700 1 -- 172.21.15.67:0/1339027081 shutdown_connections 2024-05-03T17:23:08.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.661+0000 7f078e182700 1 --2- 172.21.15.67:0/1339027081 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078810fed0 0x7f07881122c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.661+0000 7f078e182700 1 --2- 172.21.15.67:0/1339027081 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07880756d0 0x7f0788075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.661+0000 7f078e182700 1 --2- 172.21.15.67:0/1339027081 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0788075ff0 0x7f078810f990 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.661+0000 7f078e182700 1 -- 172.21.15.67:0/1339027081 >> 172.21.15.67:0/1339027081 conn(0x7f078807a610 msgr2=0x7f078807aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.661+0000 7f078e182700 1 -- 172.21.15.67:0/1339027081 shutdown_connections 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.661+0000 7f078e182700 1 -- 172.21.15.67:0/1339027081 wait complete. 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.662+0000 7f078e182700 1 Processor -- start 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.662+0000 7f078e182700 1 -- start start 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f078e182700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07880756d0 0x7f07881a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f078e182700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0788075ff0 0x7f07881a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f078e182700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078810fed0 0x7f078819bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f078e182700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f07881158a0 con 0x7f0788075ff0 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f078e182700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f0788115720 con 0x7f078810fed0 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f078e182700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f0788115a20 con 0x7f07880756d0 2024-05-03T17:23:08.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f07877fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07880756d0 0x7f07881a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f07877fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07880756d0 0x7f07881a1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:35502/0 (socket says 172.21.15.67:35502) 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f07877fe700 1 -- 172.21.15.67:0/3029727125 learned_addr learned my addr 172.21.15.67:0/3029727125 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f0787fff700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078810fed0 0x7f078819bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f0786ffd700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0788075ff0 0x7f07881a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f07877fe700 1 -- 172.21.15.67:0/3029727125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078810fed0 msgr2=0x7f078819bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f07877fe700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078810fed0 0x7f078819bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f07877fe700 1 -- 172.21.15.67:0/3029727125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0788075ff0 msgr2=0x7f07881a1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f07877fe700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0788075ff0 0x7f07881a1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f07877fe700 1 -- 172.21.15.67:0/3029727125 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078819c440 con 0x7f07880756d0 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.663+0000 7f07877fe700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07880756d0 0x7f07881a1a40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f077800e9c0 tx=0x7f077800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.664+0000 7f0784ff9700 1 -- 172.21.15.67:0/3029727125 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077800cbd0 con 0x7f07880756d0 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.664+0000 7f078e182700 1 -- 172.21.15.67:0/3029727125 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078819c730 con 0x7f07880756d0 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.664+0000 7f0784ff9700 1 -- 172.21.15.67:0/3029727125 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f077800ee60 con 0x7f07880756d0 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.664+0000 7f0784ff9700 1 -- 172.21.15.67:0/3029727125 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07780055c0 con 0x7f07880756d0 2024-05-03T17:23:08.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.664+0000 7f078e182700 1 -- 172.21.15.67:0/3029727125 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07881a89c0 con 0x7f07880756d0 2024-05-03T17:23:08.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.666+0000 7f0784ff9700 1 -- 172.21.15.67:0/3029727125 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f0778004750 con 0x7f07880756d0 2024-05-03T17:23:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.667+0000 7f0784ff9700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f07700b2910 0x7f07700b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.667+0000 7f0786ffd700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f07700b2910 0x7f07700b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.667+0000 7f0784ff9700 1 -- 172.21.15.67:0/3029727125 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f07780e2690 con 0x7f07880756d0 2024-05-03T17:23:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.667+0000 7f0798a15b80 1 Processor -- start 2024-05-03T17:23:08.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.667+0000 7f0798a15b80 1 -- start start 2024-05-03T17:23:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f0798a15b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c721740 0x561d7c721b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f0798a15b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c722060 0x561d7c71cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f0798a15b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 0x561d7c71d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f0798a15b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561d7c66f8f0 con 0x561d7c71d480 2024-05-03T17:23:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f0798a15b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561d7c5d4d50 con 0x561d7c721740 2024-05-03T17:23:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f0798a15b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x561d7c66b8b0 con 0x561d7c722060 2024-05-03T17:23:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f0786ffd700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f07700b2910 0x7f07700b4dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f077c006010 tx=0x7f077c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f0787fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 0x561d7c71d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f0787fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 0x561d7c71d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50276/0 (socket says 172.21.15.67:50276) 2024-05-03T17:23:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f0787fff700 1 -- 172.21.15.67:0/1916071263 learned_addr learned my addr 172.21.15.67:0/1916071263 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:08.674 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f07877fe700 1 --2- 172.21.15.67:0/1916071263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c721740 0x561d7c721b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f0786ffd700 1 --2- 172.21.15.67:0/1916071263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c722060 0x561d7c71cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f07877fe700 1 -- 172.21.15.67:0/1916071263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c722060 msgr2=0x561d7c71cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f07877fe700 1 --2- 172.21.15.67:0/1916071263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c722060 0x561d7c71cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f07877fe700 1 -- 172.21.15.67:0/1916071263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 msgr2=0x561d7c71d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f07877fe700 1 --2- 172.21.15.67:0/1916071263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 0x561d7c71d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f07877fe700 1 -- 172.21.15.67:0/1916071263 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d7c5bc880 con 0x561d7c721740 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f0787fff700 1 --2- 172.21.15.67:0/1916071263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 0x561d7c71d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.668+0000 7f07877fe700 1 --2- 172.21.15.67:0/1916071263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c721740 0x561d7c721b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f07780dffd0 tx=0x7f07780a9250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=963f533b55e7e75c server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.669+0000 7f076dffb700 1 -- 172.21.15.67:0/1916071263 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07780c3070 con 0x561d7c721740 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.669+0000 7f076dffb700 1 -- 172.21.15.67:0/1916071263 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07780a95f0 con 0x561d7c721740 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.669+0000 7f076dffb700 1 -- 172.21.15.67:0/1916071263 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07780a9fa0 con 0x561d7c721740 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.669+0000 7f0798a15b80 1 -- 172.21.15.67:0/1916071263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c721740 msgr2=0x561d7c721b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.669+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1916071263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c721740 0x561d7c721b20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f07780dffd0 tx=0x7f07780a9250 comp rx=0 tx=0).stop 2024-05-03T17:23:08.675 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.669+0000 7f0798a15b80 1 -- 172.21.15.67:0/1916071263 shutdown_connections 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.669+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1916071263 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 0x561d7c71d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.669+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1916071263 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c721740 0x561d7c721b20 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.669+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1916071263 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c722060 0x561d7c71cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.669+0000 7f0798a15b80 1 -- 172.21.15.67:0/1916071263 >> 172.21.15.67:0/1916071263 conn(0x561d7c649b20 msgr2=0x561d7c7179a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.669+0000 7f0798a15b80 1 -- 172.21.15.67:0/1916071263 shutdown_connections 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.669+0000 7f0798a15b80 1 -- 172.21.15.67:0/1916071263 wait complete. 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.670+0000 7f0798a15b80 1 -- start start 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.670+0000 7f0798a15b80 1 Processor -- start 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.670+0000 7f0798a15b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 0x561d7c728e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.670+0000 7f0798a15b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c721740 0x561d7c726230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.670+0000 7f0798a15b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c722060 0x561d7c726770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.670+0000 7f0798a15b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561d7c729ba0 con 0x561d7c71d480 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.670+0000 7f0798a15b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561d7c628e80 con 0x561d7c721740 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.670+0000 7f0798a15b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x561d7c729d20 con 0x561d7c722060 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.670+0000 7f0798a15b80 10 client.0 _ll_register_callbacks cb 0x561d7c603560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T17:23:08.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.670+0000 7f07877fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 0x561d7c728e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.670+0000 7f07877fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 0x561d7c728e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50278/0 (socket says 172.21.15.67:50278) 2024-05-03T17:23:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.670+0000 7f07877fe700 1 -- 172.21.15.67:0/1585963916 learned_addr learned my addr 172.21.15.67:0/1585963916 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f0786ffd700 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c721740 0x561d7c726230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f0787fff700 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c722060 0x561d7c726770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f07877fe700 1 -- 172.21.15.67:0/1585963916 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c722060 msgr2=0x561d7c726770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f07877fe700 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c722060 0x561d7c726770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f07877fe700 1 -- 172.21.15.67:0/1585963916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c721740 msgr2=0x561d7c726230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f07877fe700 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c721740 0x561d7c726230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f07877fe700 1 -- 172.21.15.67:0/1585963916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d7c626710 con 0x561d7c71d480 2024-05-03T17:23:08.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f07877fe700 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 0x561d7c728e20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f07780a6d50 tx=0x7f07780b2760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc9533f2dd0598 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f0766ffd700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T17:23:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f0766ffd700 1 -- 172.21.15.67:0/1585963916 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077806d070 con 0x561d7c71d480 2024-05-03T17:23:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f0766ffd700 1 -- 172.21.15.67:0/1585963916 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f07780a8f50 con 0x561d7c71d480 2024-05-03T17:23:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f0798a15b80 1 -- 172.21.15.67:0/1585963916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f07780a95f0 con 0x561d7c71d480 2024-05-03T17:23:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f0766ffd700 1 -- 172.21.15.67:0/1585963916 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f07780c3070 con 0x561d7c71d480 2024-05-03T17:23:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f0798a15b80 10 client.14967 fetch_fsmap learned FSMap version 25 2024-05-03T17:23:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f0798a15b80 1 -- 172.21.15.67:0/1585963916 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561d7c734460 con 0x561d7c71d480 2024-05-03T17:23:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.671+0000 7f0766ffd700 1 -- 172.21.15.67:0/1585963916 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07780a95f0 con 0x561d7c71d480 2024-05-03T17:23:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.672+0000 7f0766ffd700 1 -- 172.21.15.67:0/1585963916 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f0778087020 con 0x561d7c71d480 2024-05-03T17:23:08.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.672+0000 7f0798a15b80 10 client.14967 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T17:23:08.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.672+0000 7f0798a15b80 10 client.14967 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-03T17:23:08.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.672+0000 7f0798a15b80 20 client.14967 populate_metadata read hostname 'smithi067' 2024-05-03T17:23:08.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.672+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x561d7c7363c0 0x561d7c737830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.672+0000 7f0798a15b80 4 client.14967 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-03T17:23:08.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.672+0000 7f0798a15b80 1 -- 172.21.15.67:0/1585963916 --> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561d7c67d2d0 con 0x561d7c7363c0 2024-05-03T17:23:08.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.672+0000 7f0786ffd700 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x561d7c7363c0 0x561d7c737830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.672+0000 7f0786ffd700 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x561d7c7363c0 0x561d7c737830 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fea4aa295eb043d5 server_cookie=93d42ec7cf4840fe in_seq=0 out_seq=0 2024-05-03T17:23:08.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.672+0000 7f0766ffd700 10 client.14967 ms_handle_connect on v2:172.21.15.70:6838/902276547 2024-05-03T17:23:08.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0766ffd700 1 -- 172.21.15.67:0/1585963916 <== mds.0 v2:172.21.15.70:6838/902276547 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561d7c67d2d0 con 0x561d7c7363c0 2024-05-03T17:23:08.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0766ffd700 10 client.14967 handle_command_reply: tid=0 2024-05-03T17:23:08.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0798a15b80 1 client.14967 shutdown 2024-05-03T17:23:08.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0798a15b80 1 -- 172.21.15.67:0/1585963916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 msgr2=0x561d7c728e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:08.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 0x561d7c728e20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f07780a6d50 tx=0x7f07780b2760 comp rx=0 tx=0).stop 2024-05-03T17:23:08.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0798a15b80 1 -- 172.21.15.67:0/1585963916 shutdown_connections 2024-05-03T17:23:08.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x561d7c7363c0 0x561d7c737830 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c722060 0x561d7c726770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0766ffd700 0 client.14967 ms_handle_reset on v2:172.21.15.70:6838/902276547 2024-05-03T17:23:08.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c71d480 0x561d7c728e20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1585963916 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c721740 0x561d7c726230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0798a15b80 1 -- 172.21.15.67:0/1585963916 >> 172.21.15.67:0/1585963916 conn(0x561d7c649b20 msgr2=0x561d7c716800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:08.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0798a15b80 1 -- 172.21.15.67:0/1585963916 shutdown_connections 2024-05-03T17:23:08.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.673+0000 7f0798a15b80 1 -- 172.21.15.67:0/1585963916 wait complete. 2024-05-03T17:23:08.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.674+0000 7f0798a15b80 20 client.14967 trim_cache size 0 max 16384 2024-05-03T17:23:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f0798a15b80 1 Processor -- start 2024-05-03T17:23:08.697 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f0798a15b80 1 -- start start 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f0798a15b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c649b20 0x561d7c7326c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f0798a15b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c71d480 0x561d7c732c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f0798a15b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c721740 0x561d7c740ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f0798a15b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561d7c735b70 con 0x561d7c721740 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f0798a15b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561d7c736240 con 0x561d7c649b20 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f0798a15b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x561d7c736910 con 0x561d7c71d480 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f0787fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c721740 0x561d7c740ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f0787fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c721740 0x561d7c740ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50288/0 (socket says 172.21.15.67:50288) 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f0787fff700 1 -- 172.21.15.67:0/344460901 learned_addr learned my addr 172.21.15.67:0/344460901 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f0786ffd700 1 --2- 172.21.15.67:0/344460901 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c71d480 0x561d7c732c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.695+0000 7f07877fe700 1 --2- 172.21.15.67:0/344460901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c649b20 0x561d7c7326c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.696+0000 7f0787fff700 1 -- 172.21.15.67:0/344460901 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c71d480 msgr2=0x561d7c732c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.696+0000 7f0787fff700 1 --2- 172.21.15.67:0/344460901 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c71d480 0x561d7c732c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.698 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.696+0000 7f0787fff700 1 -- 172.21.15.67:0/344460901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c649b20 msgr2=0x561d7c7326c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.696+0000 7f0787fff700 1 --2- 172.21.15.67:0/344460901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c649b20 0x561d7c7326c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.696+0000 7f0787fff700 1 -- 172.21.15.67:0/344460901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561d7c7247a0 con 0x561d7c721740 2024-05-03T17:23:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.696+0000 7f0787fff700 1 --2- 172.21.15.67:0/344460901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c721740 0x561d7c740ca0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0780014750 tx=0x7f0780007950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf6d01539d332fd4 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.696+0000 7f076d7fa700 1 -- 172.21.15.67:0/344460901 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0780004bb0 con 0x561d7c721740 2024-05-03T17:23:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.696+0000 7f076d7fa700 1 -- 172.21.15.67:0/344460901 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0780004d50 con 0x561d7c721740 2024-05-03T17:23:08.699 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.696+0000 7f076d7fa700 1 -- 172.21.15.67:0/344460901 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0780005740 con 0x561d7c721740 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.697+0000 7f0798a15b80 1 -- 172.21.15.67:0/344460901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c721740 msgr2=0x561d7c740ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.697+0000 7f0798a15b80 1 --2- 172.21.15.67:0/344460901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c721740 0x561d7c740ca0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0780014750 tx=0x7f0780007950 comp rx=0 tx=0).stop 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.697+0000 7f0798a15b80 1 -- 172.21.15.67:0/344460901 shutdown_connections 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.697+0000 7f0798a15b80 1 --2- 172.21.15.67:0/344460901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c721740 0x561d7c740ca0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.697+0000 7f0798a15b80 1 --2- 172.21.15.67:0/344460901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c649b20 0x561d7c7326c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.697+0000 7f0798a15b80 1 --2- 172.21.15.67:0/344460901 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c71d480 0x561d7c732c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.697+0000 7f0798a15b80 1 -- 172.21.15.67:0/344460901 >> 172.21.15.67:0/344460901 conn(0x561d7c7143c0 msgr2=0x561d7c731fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.697+0000 7f0798a15b80 1 -- 172.21.15.67:0/344460901 shutdown_connections 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.697+0000 7f0798a15b80 1 -- 172.21.15.67:0/344460901 wait complete. 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.697+0000 7f0798a15b80 1 -- start start 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.697+0000 7f0798a15b80 1 Processor -- start 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0798a15b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c649b20 0x561d7c73da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.703 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0798a15b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c71d480 0x561d7c73df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0798a15b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c721740 0x561d7c73e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0798a15b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x561d7c7376b0 con 0x561d7c649b20 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0798a15b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x561d7c736fe0 con 0x561d7c71d480 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0798a15b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x561d7c737d80 con 0x561d7c721740 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0798a15b80 10 client.0 _ll_register_callbacks cb 0x561d7c603560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f07877fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c649b20 0x561d7c73da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f07877fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c649b20 0x561d7c73da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50298/0 (socket says 172.21.15.67:50298) 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f07877fe700 1 -- 172.21.15.67:0/1133105200 learned_addr learned my addr 172.21.15.67:0/1133105200 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0787fff700 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c721740 0x561d7c73e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0786ffd700 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c71d480 0x561d7c73df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f07877fe700 1 -- 172.21.15.67:0/1133105200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c721740 msgr2=0x561d7c73e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f07877fe700 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c721740 0x561d7c73e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f07877fe700 1 -- 172.21.15.67:0/1133105200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c71d480 msgr2=0x561d7c73df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f07877fe700 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c71d480 0x561d7c73df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.704 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f07877fe700 1 -- 172.21.15.67:0/1133105200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561d7c7a1aa0 con 0x561d7c649b20 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f07877fe700 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c649b20 0x561d7c73da20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f077806f310 tx=0x7f0778005bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe424b896c0cf09 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0766ffd700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0766ffd700 1 -- 172.21.15.67:0/1133105200 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077806d040 con 0x561d7c649b20 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0766ffd700 1 -- 172.21.15.67:0/1133105200 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f07780a5570 con 0x561d7c649b20 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.698+0000 7f0798a15b80 1 -- 172.21.15.67:0/1133105200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0780004d50 con 0x561d7c649b20 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.699+0000 7f0766ffd700 1 -- 172.21.15.67:0/1133105200 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f07780a9250 con 0x561d7c649b20 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.699+0000 7f0798a15b80 10 client.14973 fetch_fsmap learned FSMap version 25 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.699+0000 7f0798a15b80 1 -- 172.21.15.67:0/1133105200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x561d7c7a2d40 con 0x561d7c649b20 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.699+0000 7f0766ffd700 1 -- 172.21.15.67:0/1133105200 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0780004d50 con 0x561d7c649b20 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.699+0000 7f0766ffd700 1 -- 172.21.15.67:0/1133105200 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f0778087040 con 0x561d7c649b20 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.699+0000 7f0798a15b80 10 client.14973 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.699+0000 7f0798a15b80 10 client.14973 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.699+0000 7f0798a15b80 20 client.14973 populate_metadata read hostname 'smithi067' 2024-05-03T17:23:08.705 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.699+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x561d7c722060 0x561d7c7a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.699+0000 7f0798a15b80 4 client.14973 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.699+0000 7f0798a15b80 1 -- 172.21.15.67:0/1133105200 --> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] -- command(tid 0: {"prefix": "status"}) -- 0x561d7c677370 con 0x561d7c722060 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.700+0000 7f0786ffd700 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x561d7c722060 0x561d7c7a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.700+0000 7f0786ffd700 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x561d7c722060 0x561d7c7a63b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83edbcbfca50e0b9 server_cookie=d178288d097f471c in_seq=0 out_seq=0 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.700+0000 7f0766ffd700 10 client.14973 ms_handle_connect on v2:172.21.15.70:6838/902276547 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0766ffd700 1 -- 172.21.15.67:0/1133105200 <== mds.0 v2:172.21.15.70:6838/902276547 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x561d7c677370 con 0x561d7c722060 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0766ffd700 10 client.14973 handle_command_reply: tid=0 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0798a15b80 1 client.14973 shutdown 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0798a15b80 1 -- 172.21.15.67:0/1133105200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c649b20 msgr2=0x561d7c73da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c649b20 0x561d7c73da20 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f077806f310 tx=0x7f0778005bb0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0798a15b80 1 -- 172.21.15.67:0/1133105200 shutdown_connections 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x561d7c722060 0x561d7c7a63b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x561d7c721740 0x561d7c73e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x561d7c649b20 0x561d7c73da20 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0798a15b80 1 --2- 172.21.15.67:0/1133105200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x561d7c71d480 0x561d7c73df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0798a15b80 1 -- 172.21.15.67:0/1133105200 >> 172.21.15.67:0/1133105200 conn(0x561d7c7143c0 msgr2=0x561d7c732840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0766ffd700 0 client.14973 ms_handle_reset on v2:172.21.15.70:6838/902276547 2024-05-03T17:23:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0798a15b80 1 -- 172.21.15.67:0/1133105200 shutdown_connections 2024-05-03T17:23:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0798a15b80 1 -- 172.21.15.67:0/1133105200 wait complete. 2024-05-03T17:23:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f0798a15b80 20 client.14973 trim_cache size 0 max 16384 2024-05-03T17:23:08.707 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T17:23:08.707 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "20c8e22e-0970-11ef-bc97-c7b262605968", 2024-05-03T17:23:08.707 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-05-03T17:23:08.707 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24464, 2024-05-03T17:23:08.707 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-03T17:23:08.708 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-03T17:23:08.708 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-03T17:23:08.708 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 148.49725518299999, 2024-05-03T17:23:08.708 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-03T17:23:08.708 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 81, 2024-05-03T17:23:08.708 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 79, 2024-05-03T17:23:08.708 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 165.49962309 2024-05-03T17:23:08.708 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T17:23:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f078e182700 1 -- 172.21.15.67:0/3029727125 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f07700b2910 msgr2=0x7f07700b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f078e182700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f07700b2910 0x7f07700b4dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f077c006010 tx=0x7f077c005960 comp rx=0 tx=0).stop 2024-05-03T17:23:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f078e182700 1 -- 172.21.15.67:0/3029727125 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07880756d0 msgr2=0x7f07881a1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.701+0000 7f078e182700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07880756d0 0x7f07881a1a40 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f077800e9c0 tx=0x7f077800ecd0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.702+0000 7f078e182700 1 -- 172.21.15.67:0/3029727125 shutdown_connections 2024-05-03T17:23:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.702+0000 7f078e182700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f07700b2910 0x7f07700b4dd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.702+0000 7f078e182700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f078810fed0 0x7f078819bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.702+0000 7f078e182700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f07880756d0 0x7f07881a1a40 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.702+0000 7f078e182700 1 --2- 172.21.15.67:0/3029727125 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0788075ff0 0x7f07881a1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.702+0000 7f078e182700 1 -- 172.21.15.67:0/3029727125 >> 172.21.15.67:0/3029727125 conn(0x7f078807a610 msgr2=0x7f07881006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.702+0000 7f078e182700 1 -- 172.21.15.67:0/3029727125 shutdown_connections 2024-05-03T17:23:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:08.702+0000 7f078e182700 1 -- 172.21.15.67:0/3029727125 wait complete. 2024-05-03T17:23:09.556 INFO:tasks.cephadm:Setup complete, yielding 2024-05-03T17:23:09.557 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-03T17:23:09.572 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi067.front.sepia.ceph.com 2024-05-03T17:23:09.573 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- bash -c 'ceph orch status' 2024-05-03T17:23:09.837 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:09.859 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:09 smithi067 ceph-mon[29731]: pgmap v372: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:10.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:09 smithi082 ceph-mon[33535]: pgmap v372: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:10.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:09 smithi070 ceph-mon[33250]: pgmap v372: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:11.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.638+0000 7f8e11cdb700 1 -- 172.21.15.67:0/1601333949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c104700 msgr2=0x7f8e0c104b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:11.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.638+0000 7f8e11cdb700 1 --2- 172.21.15.67:0/1601333949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c104700 0x7f8e0c104b60 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f8e0400b0d0 tx=0x7f8e0400b3e0 comp rx=0 tx=0).stop 2024-05-03T17:23:11.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.638+0000 7f8e11cdb700 1 -- 172.21.15.67:0/1601333949 shutdown_connections 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.638+0000 7f8e11cdb700 1 --2- 172.21.15.67:0/1601333949 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e0c1050a0 0x7f8e0c10b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.638+0000 7f8e11cdb700 1 --2- 172.21.15.67:0/1601333949 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e0c103500 0x7f8e0c103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.638+0000 7f8e11cdb700 1 --2- 172.21.15.67:0/1601333949 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c104700 0x7f8e0c104b60 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.638+0000 7f8e11cdb700 1 -- 172.21.15.67:0/1601333949 >> 172.21.15.67:0/1601333949 conn(0x7f8e0c0feaa0 msgr2=0x7f8e0c100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.638+0000 7f8e11cdb700 1 -- 172.21.15.67:0/1601333949 shutdown_connections 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.638+0000 7f8e11cdb700 1 -- 172.21.15.67:0/1601333949 wait complete. 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e11cdb700 1 Processor -- start 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e11cdb700 1 -- start start 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e11cdb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c103500 0x7f8e0c077240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e11cdb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e0c104700 0x7f8e0c075890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e11cdb700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e0c1050a0 0x7f8e0c075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e11cdb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c10ef80 con 0x7f8e0c103500 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e11cdb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c10ee00 con 0x7f8e0c104700 2024-05-03T17:23:11.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e11cdb700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8e0c10f100 con 0x7f8e0c1050a0 2024-05-03T17:23:11.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e0bfff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e0c1050a0 0x7f8e0c075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:11.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e0affd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e0c104700 0x7f8e0c075890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:11.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e0b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c103500 0x7f8e0c077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:11.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e0bfff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e0c1050a0 0x7f8e0c075dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:35568/0 (socket says 172.21.15.67:35568) 2024-05-03T17:23:11.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.639+0000 7f8e0bfff700 1 -- 172.21.15.67:0/4219331345 learned_addr learned my addr 172.21.15.67:0/4219331345 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:11.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.640+0000 7f8e0bfff700 1 -- 172.21.15.67:0/4219331345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e0c104700 msgr2=0x7f8e0c075890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:11.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.640+0000 7f8e0bfff700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e0c104700 0x7f8e0c075890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:11.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.640+0000 7f8e0bfff700 1 -- 172.21.15.67:0/4219331345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c103500 msgr2=0x7f8e0c077240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:11.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.640+0000 7f8e0bfff700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c103500 0x7f8e0c077240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:11.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.640+0000 7f8e0bfff700 1 -- 172.21.15.67:0/4219331345 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e0c076680 con 0x7f8e0c1050a0 2024-05-03T17:23:11.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.640+0000 7f8e0b7fe700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c103500 0x7f8e0c077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:23:11.643 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.640+0000 7f8e0bfff700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e0c1050a0 0x7f8e0c075dd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000d980 tx=0x7f8e0000dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:11.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.640+0000 7f8e08ff9700 1 -- 172.21.15.67:0/4219331345 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e00017070 con 0x7f8e0c1050a0 2024-05-03T17:23:11.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.640+0000 7f8e11cdb700 1 -- 172.21.15.67:0/4219331345 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e0c1a37c0 con 0x7f8e0c1050a0 2024-05-03T17:23:11.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.640+0000 7f8e08ff9700 1 -- 172.21.15.67:0/4219331345 <== mon.1 v2:172.21.15.82:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e00005020 con 0x7f8e0c1050a0 2024-05-03T17:23:11.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.640+0000 7f8e08ff9700 1 -- 172.21.15.67:0/4219331345 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e0000fa10 con 0x7f8e0c1050a0 2024-05-03T17:23:11.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.640+0000 7f8e11cdb700 1 -- 172.21.15.67:0/4219331345 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e0c1a3cd0 con 0x7f8e0c1050a0 2024-05-03T17:23:11.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.641+0000 7f8e11cdb700 1 -- 172.21.15.67:0/4219331345 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e0c06a190 con 0x7f8e0c1050a0 2024-05-03T17:23:11.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.643+0000 7f8e08ff9700 1 -- 172.21.15.67:0/4219331345 <== mon.1 v2:172.21.15.82:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f8e00004a30 con 0x7f8e0c1050a0 2024-05-03T17:23:11.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.643+0000 7f8e08ff9700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8df40b2990 0x7f8df40b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:11.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.644+0000 7f8e0b7fe700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8df40b2990 0x7f8df40b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:11.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.644+0000 7f8e08ff9700 1 -- 172.21.15.67:0/4219331345 <== mon.1 v2:172.21.15.82:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f8e000e2600 con 0x7f8e0c1050a0 2024-05-03T17:23:11.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.644+0000 7f8e0b7fe700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8df40b2990 0x7f8df40b4e50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c104560 tx=0x7f8dfc007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:11.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.647+0000 7f8e08ff9700 1 -- 172.21.15.67:0/4219331345 <== mon.1 v2:172.21.15.82:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f8e00020020 con 0x7f8e0c1050a0 2024-05-03T17:23:11.882 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.881+0000 7f8e11cdb700 1 -- 172.21.15.67:0/4219331345 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f8e0c0638e0 con 0x7f8df40b2990 2024-05-03T17:23:11.883 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.882+0000 7f8e08ff9700 1 -- 172.21.15.67:0/4219331345 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f8e0c0638e0 con 0x7f8df40b2990 2024-05-03T17:23:11.883 INFO:teuthology.orchestra.run.smithi067.stdout:Backend: cephadm 2024-05-03T17:23:11.884 INFO:teuthology.orchestra.run.smithi067.stdout:Available: Yes 2024-05-03T17:23:11.884 INFO:teuthology.orchestra.run.smithi067.stdout:Paused: No 2024-05-03T17:23:11.885 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.885+0000 7f8e11cdb700 1 -- 172.21.15.67:0/4219331345 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8df40b2990 msgr2=0x7f8df40b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:11.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.885+0000 7f8e11cdb700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8df40b2990 0x7f8df40b4e50 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c104560 tx=0x7f8dfc007400 comp rx=0 tx=0).stop 2024-05-03T17:23:11.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.885+0000 7f8e11cdb700 1 -- 172.21.15.67:0/4219331345 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e0c1050a0 msgr2=0x7f8e0c075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:11.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.885+0000 7f8e11cdb700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e0c1050a0 0x7f8e0c075dd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8e0000d980 tx=0x7f8e0000dc90 comp rx=0 tx=0).stop 2024-05-03T17:23:11.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.885+0000 7f8e11cdb700 1 -- 172.21.15.67:0/4219331345 shutdown_connections 2024-05-03T17:23:11.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.885+0000 7f8e11cdb700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8df40b2990 0x7f8df40b4e50 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:11.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.885+0000 7f8e11cdb700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8e0c1050a0 0x7f8e0c075dd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:11.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.885+0000 7f8e11cdb700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8e0c103500 0x7f8e0c077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:11.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.885+0000 7f8e11cdb700 1 --2- 172.21.15.67:0/4219331345 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8e0c104700 0x7f8e0c075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:11.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.885+0000 7f8e11cdb700 1 -- 172.21.15.67:0/4219331345 >> 172.21.15.67:0/4219331345 conn(0x7f8e0c0feaa0 msgr2=0x7f8e0c107a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:11.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.885+0000 7f8e11cdb700 1 -- 172.21.15.67:0/4219331345 shutdown_connections 2024-05-03T17:23:11.886 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:11.885+0000 7f8e11cdb700 1 -- 172.21.15.67:0/4219331345 wait complete. 2024-05-03T17:23:11.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:11 smithi067 ceph-mon[29731]: pgmap v373: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:12.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:11 smithi082 ceph-mon[33535]: pgmap v373: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:12.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:11 smithi070 ceph-mon[33250]: pgmap v373: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:12.720 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-03T17:23:12.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:12 smithi067 ceph-mon[29731]: from='client.24796 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:12.975 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:13.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:12 smithi082 ceph-mon[33535]: from='client.24796 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:13.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:12 smithi070 ceph-mon[33250]: from='client.24796 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:13.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:13 smithi067 ceph-mon[29731]: pgmap v374: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:14.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:13 smithi082 ceph-mon[33535]: pgmap v374: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:13 smithi070 ceph-mon[33250]: pgmap v374: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.705+0000 7fe835246700 1 -- 172.21.15.67:0/195549376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe830103500 msgr2=0x7fe830103920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.705+0000 7fe835246700 1 --2- 172.21.15.67:0/195549376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe830103500 0x7fe830103920 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe8200099c0 tx=0x7fe820009cd0 comp rx=0 tx=0).stop 2024-05-03T17:23:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.705+0000 7fe835246700 1 -- 172.21.15.67:0/195549376 shutdown_connections 2024-05-03T17:23:14.706 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.705+0000 7fe835246700 1 --2- 172.21.15.67:0/195549376 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8301050a0 0x7fe83010b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:14.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.705+0000 7fe835246700 1 --2- 172.21.15.67:0/195549376 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe830103500 0x7fe830103920 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:14.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.705+0000 7fe835246700 1 --2- 172.21.15.67:0/195549376 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe830104700 0x7fe830104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:14.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.705+0000 7fe835246700 1 -- 172.21.15.67:0/195549376 >> 172.21.15.67:0/195549376 conn(0x7fe8300feaa0 msgr2=0x7fe830100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:14.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.705+0000 7fe835246700 1 -- 172.21.15.67:0/195549376 shutdown_connections 2024-05-03T17:23:14.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.705+0000 7fe835246700 1 -- 172.21.15.67:0/195549376 wait complete. 2024-05-03T17:23:14.707 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.706+0000 7fe835246700 1 Processor -- start 2024-05-03T17:23:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.706+0000 7fe835246700 1 -- start start 2024-05-03T17:23:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.706+0000 7fe835246700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe830103500 0x7fe830199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:14.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.706+0000 7fe835246700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe830104700 0x7fe830199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.706+0000 7fe835246700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8301050a0 0x7fe8301a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.706+0000 7fe835246700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe83010f070 con 0x7fe8301050a0 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.706+0000 7fe835246700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe83010eef0 con 0x7fe830104700 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.706+0000 7fe835246700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe83010f1f0 con 0x7fe830103500 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe82f59e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8301050a0 0x7fe8301a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe82f59e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8301050a0 0x7fe8301a0d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:50340/0 (socket says 172.21.15.67:50340) 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe82f59e700 1 -- 172.21.15.67:0/1358476946 learned_addr learned my addr 172.21.15.67:0/1358476946 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe82ed9d700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe830103500 0x7fe830199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe82e59c700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe830104700 0x7fe830199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe82f59e700 1 -- 172.21.15.67:0/1358476946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe830103500 msgr2=0x7fe830199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe82f59e700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe830103500 0x7fe830199760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe82f59e700 1 -- 172.21.15.67:0/1358476946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe830104700 msgr2=0x7fe830199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe82f59e700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe830104700 0x7fe830199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe82f59e700 1 -- 172.21.15.67:0/1358476946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8301a13c0 con 0x7fe8301050a0 2024-05-03T17:23:14.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe82f59e700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8301050a0 0x7fe8301a0d20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe82800b9a0 tx=0x7fe82800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe817fff700 1 -- 172.21.15.67:0/1358476946 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82800d850 con 0x7fe8301050a0 2024-05-03T17:23:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe835246700 1 -- 172.21.15.67:0/1358476946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8301a16b0 con 0x7fe8301050a0 2024-05-03T17:23:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe817fff700 1 -- 172.21.15.67:0/1358476946 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe82800e420 con 0x7fe8301050a0 2024-05-03T17:23:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe817fff700 1 -- 172.21.15.67:0/1358476946 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe828017640 con 0x7fe8301050a0 2024-05-03T17:23:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.707+0000 7fe835246700 1 -- 172.21.15.67:0/1358476946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8301a1c70 con 0x7fe8301050a0 2024-05-03T17:23:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.709+0000 7fe835246700 1 -- 172.21.15.67:0/1358476946 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe83006a190 con 0x7fe8301050a0 2024-05-03T17:23:14.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.710+0000 7fe817fff700 1 -- 172.21.15.67:0/1358476946 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fe8280203f0 con 0x7fe8301050a0 2024-05-03T17:23:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.710+0000 7fe817fff700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe8180b2910 0x7fe8180b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.711+0000 7fe82ed9d700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe8180b2910 0x7fe8180b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.711+0000 7fe817fff700 1 -- 172.21.15.67:0/1358476946 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fe8280e1f00 con 0x7fe8301050a0 2024-05-03T17:23:14.711 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.711+0000 7fe82ed9d700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe8180b2910 0x7fe8180b4dd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe8200038c0 tx=0x7fe82000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:14.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.715+0000 7fe817fff700 1 -- 172.21.15.67:0/1358476946 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe828013030 con 0x7fe8301050a0 2024-05-03T17:23:14.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.936+0000 7fe835246700 1 -- 172.21.15.67:0/1358476946 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fe8300638e0 con 0x7fe8180b2910 2024-05-03T17:23:14.950 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.950+0000 7fe817fff700 1 -- 172.21.15.67:0/1358476946 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7fe8300638e0 con 0x7fe8180b2910 2024-05-03T17:23:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-03T17:23:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:mds.a smithi067 running (2m) 32s ago 2m 14.4M - 19.0.0-3450-g9a047fe7 ee4d84620249 5f4389aa2a4a 2024-05-03T17:23:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:mds.b smithi070 running (2m) 36s ago 2m 25.7M - 19.0.0-3450-g9a047fe7 ee4d84620249 d4ff0e3af46a 2024-05-03T17:23:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:mds.c smithi082 running (3m) 36s ago 3m 20.3M - 19.0.0-3450-g9a047fe7 ee4d84620249 664901c28883 2024-05-03T17:23:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:mds.d smithi067 running (3m) 32s ago 3m 14.3M - 19.0.0-3450-g9a047fe7 ee4d84620249 3829824985cb 2024-05-03T17:23:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:mds.e smithi070 running (3m) 36s ago 3m 20.6M - 19.0.0-3450-g9a047fe7 ee4d84620249 53db2ce50df7 2024-05-03T17:23:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:mds.f smithi082 running (3m) 36s ago 3m 20.6M - 19.0.0-3450-g9a047fe7 ee4d84620249 18fdab0f4e0f 2024-05-03T17:23:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:mds.g smithi067 running (2m) 32s ago 2m 14.5M - 19.0.0-3450-g9a047fe7 ee4d84620249 08c0c8ab3a74 2024-05-03T17:23:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:mds.h smithi070 running (3m) 36s ago 3m 21.3M - 19.0.0-3450-g9a047fe7 ee4d84620249 6563c074bc62 2024-05-03T17:23:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:mds.i smithi082 running (3m) 36s ago 3m 21.0M - 19.0.0-3450-g9a047fe7 ee4d84620249 e05286d474b3 2024-05-03T17:23:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:mds.j smithi067 running (3m) 32s ago 3m 14.6M - 19.0.0-3450-g9a047fe7 ee4d84620249 3d98710bbba2 2024-05-03T17:23:14.951 INFO:teuthology.orchestra.run.smithi067.stdout:mds.k smithi070 running (3m) 36s ago 3m 16.0M - 19.0.0-3450-g9a047fe7 ee4d84620249 e4257f734574 2024-05-03T17:23:14.952 INFO:teuthology.orchestra.run.smithi067.stdout:mds.l smithi082 running (3m) 36s ago 3m 14.4M - 19.0.0-3450-g9a047fe7 ee4d84620249 25e9b16902a5 2024-05-03T17:23:14.952 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.x smithi067 *:9283,8765 running (10m) 32s ago 10m 530M - 19.0.0-3450-g9a047fe7 ee4d84620249 0a733d8f609c 2024-05-03T17:23:14.952 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.y smithi070 *:8443,8765 running (8m) 36s ago 8m 468M - 19.0.0-3450-g9a047fe7 ee4d84620249 c09dfad8badb 2024-05-03T17:23:14.952 INFO:teuthology.orchestra.run.smithi067.stdout:mgr.z smithi082 *:8443,8765 running (8m) 36s ago 8m 468M - 19.0.0-3450-g9a047fe7 ee4d84620249 d4f248973a3e 2024-05-03T17:23:14.952 INFO:teuthology.orchestra.run.smithi067.stdout:mon.a smithi067 running (10m) 32s ago 11m 54.4M 2048M 19.0.0-3450-g9a047fe7 ee4d84620249 60217e9ce4fc 2024-05-03T17:23:14.952 INFO:teuthology.orchestra.run.smithi067.stdout:mon.b smithi070 running (9m) 36s ago 9m 39.5M 2048M 19.0.0-3450-g9a047fe7 ee4d84620249 e15cdb79ea60 2024-05-03T17:23:14.952 INFO:teuthology.orchestra.run.smithi067.stdout:mon.c smithi082 running (9m) 36s ago 9m 41.1M 2048M 19.0.0-3450-g9a047fe7 ee4d84620249 2c1215b5a586 2024-05-03T17:23:14.952 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 smithi067 running (8m) 32s ago 8m 56.5M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 225b00db2e1f 2024-05-03T17:23:14.952 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 smithi070 running (8m) 36s ago 8m 45.4M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 228df925f764 2024-05-03T17:23:14.952 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 smithi082 running (7m) 36s ago 7m 46.0M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 e284ff54fd72 2024-05-03T17:23:14.952 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 smithi067 running (7m) 32s ago 7m 54.9M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 86ee56708991 2024-05-03T17:23:14.953 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 smithi070 running (6m) 36s ago 6m 42.5M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 aee95c03016d 2024-05-03T17:23:14.953 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 smithi082 running (6m) 36s ago 6m 43.8M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 e2bb673a94c0 2024-05-03T17:23:14.953 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 smithi067 running (6m) 32s ago 6m 52.7M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 d34d397dffd1 2024-05-03T17:23:14.953 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 smithi070 running (5m) 36s ago 5m 44.0M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 3ed45e5cc559 2024-05-03T17:23:14.953 INFO:teuthology.orchestra.run.smithi067.stdout:osd.8 smithi082 running (5m) 36s ago 5m 43.6M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 c3fcbbe3a317 2024-05-03T17:23:14.953 INFO:teuthology.orchestra.run.smithi067.stdout:osd.9 smithi067 running (4m) 32s ago 4m 53.0M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 63149a62cb1b 2024-05-03T17:23:14.953 INFO:teuthology.orchestra.run.smithi067.stdout:osd.10 smithi070 running (4m) 36s ago 4m 53.4M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 0cf0da3eed51 2024-05-03T17:23:14.953 INFO:teuthology.orchestra.run.smithi067.stdout:osd.11 smithi082 running (3m) 36s ago 3m 41.4M 4272M 19.0.0-3450-g9a047fe7 ee4d84620249 71dfc6a2bf40 2024-05-03T17:23:14.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.952+0000 7fe835246700 1 -- 172.21.15.67:0/1358476946 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe8180b2910 msgr2=0x7fe8180b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:14.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.952+0000 7fe835246700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe8180b2910 0x7fe8180b4dd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe8200038c0 tx=0x7fe82000b540 comp rx=0 tx=0).stop 2024-05-03T17:23:14.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.952+0000 7fe835246700 1 -- 172.21.15.67:0/1358476946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8301050a0 msgr2=0x7fe8301a0d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:14.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.952+0000 7fe835246700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8301050a0 0x7fe8301a0d20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fe82800b9a0 tx=0x7fe82800bcb0 comp rx=0 tx=0).stop 2024-05-03T17:23:14.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.952+0000 7fe835246700 1 -- 172.21.15.67:0/1358476946 shutdown_connections 2024-05-03T17:23:14.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.952+0000 7fe835246700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe8180b2910 0x7fe8180b4dd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:14.954 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.952+0000 7fe835246700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8301050a0 0x7fe8301a0d20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.952+0000 7fe835246700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe830103500 0x7fe830199760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.952+0000 7fe835246700 1 --2- 172.21.15.67:0/1358476946 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe830104700 0x7fe830199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.952+0000 7fe835246700 1 -- 172.21.15.67:0/1358476946 >> 172.21.15.67:0/1358476946 conn(0x7fe8300feaa0 msgr2=0x7fe830107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.952+0000 7fe835246700 1 -- 172.21.15.67:0/1358476946 shutdown_connections 2024-05-03T17:23:14.955 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:14.952+0000 7fe835246700 1 -- 172.21.15.67:0/1358476946 wait complete. 2024-05-03T17:23:15.862 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:15 smithi067 ceph-mon[29731]: pgmap v375: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:15.862 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:15 smithi067 ceph-mon[29731]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:15.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-03T17:23:16.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:15 smithi082 ceph-mon[33535]: pgmap v375: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:16.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:15 smithi082 ceph-mon[33535]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:16.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:15 smithi070 ceph-mon[33250]: pgmap v375: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:16.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:15 smithi070 ceph-mon[33250]: from='client.14985 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:16.154 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:17.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.847+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/2288550133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b4103550 msgr2=0x7fe1b4103970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.847+0000 7fe1b9a3a700 1 --2- 172.21.15.67:0/2288550133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b4103550 0x7fe1b4103970 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe1a40099c0 tx=0x7fe1a4009cd0 comp rx=0 tx=0).stop 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.847+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/2288550133 shutdown_connections 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.847+0000 7fe1b9a3a700 1 --2- 172.21.15.67:0/2288550133 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b41050f0 0x7fe1b410b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.847+0000 7fe1b9a3a700 1 --2- 172.21.15.67:0/2288550133 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b4103550 0x7fe1b4103970 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.847+0000 7fe1b9a3a700 1 --2- 172.21.15.67:0/2288550133 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe1b4104750 0x7fe1b4104bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.847+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/2288550133 >> 172.21.15.67:0/2288550133 conn(0x7fe1b40feaf0 msgr2=0x7fe1b4100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.847+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/2288550133 shutdown_connections 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.848+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/2288550133 wait complete. 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.848+0000 7fe1b9a3a700 1 Processor -- start 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.848+0000 7fe1b9a3a700 1 -- start start 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.848+0000 7fe1b9a3a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b4103550 0x7fe1b4199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.848+0000 7fe1b9a3a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b4104750 0x7fe1b4199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:17.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.848+0000 7fe1b9a3a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe1b41050f0 0x7fe1b41a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.848+0000 7fe1b9a3a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b410efd0 con 0x7fe1b4103550 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.848+0000 7fe1b9a3a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b410ee50 con 0x7fe1b4104750 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.848+0000 7fe1b9a3a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b410f150 con 0x7fe1b41050f0 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b4103550 0x7fe1b4199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b4103550 0x7fe1b4199610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53108/0 (socket says 172.21.15.67:53108) 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b2ffd700 1 -- 172.21.15.67:0/380330956 learned_addr learned my addr 172.21.15.67:0/380330956 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b37fe700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe1b41050f0 0x7fe1b41a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b27fc700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b4104750 0x7fe1b4199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b2ffd700 1 -- 172.21.15.67:0/380330956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe1b41050f0 msgr2=0x7fe1b41a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b2ffd700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe1b41050f0 0x7fe1b41a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b2ffd700 1 -- 172.21.15.67:0/380330956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b4104750 msgr2=0x7fe1b4199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b2ffd700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b4104750 0x7fe1b4199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b2ffd700 1 -- 172.21.15.67:0/380330956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b41a1270 con 0x7fe1b4103550 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b27fc700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b4104750 0x7fe1b4199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:23:17.852 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b2ffd700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b4103550 0x7fe1b4199610 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009f50 tx=0x7fe1a40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b8a38700 1 -- 172.21.15.67:0/380330956 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4027070 con 0x7fe1b4103550 2024-05-03T17:23:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/380330956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b41a1500 con 0x7fe1b4103550 2024-05-03T17:23:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b8a38700 1 -- 172.21.15.67:0/380330956 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1a40040a0 con 0x7fe1b4103550 2024-05-03T17:23:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b8a38700 1 -- 172.21.15.67:0/380330956 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1a4022da0 con 0x7fe1b4103550 2024-05-03T17:23:17.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.849+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/380330956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b41a1a30 con 0x7fe1b4103550 2024-05-03T17:23:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.851+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/380330956 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1b406a190 con 0x7fe1b4103550 2024-05-03T17:23:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.853+0000 7fe1b8a38700 1 -- 172.21.15.67:0/380330956 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fe1a402a070 con 0x7fe1b4103550 2024-05-03T17:23:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.853+0000 7fe1b8a38700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe19c0b2910 0x7fe19c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.853+0000 7fe1b27fc700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe19c0b2910 0x7fe19c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.853+0000 7fe1b8a38700 1 -- 172.21.15.67:0/380330956 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fe1a40e3530 con 0x7fe1b4103550 2024-05-03T17:23:17.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.854+0000 7fe1b27fc700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe19c0b2910 0x7fe19c0b4dd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8005950 tx=0x7fe1a80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:17.857 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:17.857+0000 7fe1b8a38700 1 -- 172.21.15.67:0/380330956 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe1a4031070 con 0x7fe1b4103550 2024-05-03T17:23:17.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:17 smithi067 ceph-mon[29731]: pgmap v376: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:18.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:17 smithi082 ceph-mon[33535]: pgmap v376: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:18.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:17 smithi070 ceph-mon[33250]: pgmap v376: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:18.081 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.081+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/380330956 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fe1b40638e0 con 0x7fe19c0b2910 2024-05-03T17:23:18.087 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.087+0000 7fe1b8a38700 1 -- 172.21.15.67:0/380330956 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fe1b40638e0 con 0x7fe19c0b2910 2024-05-03T17:23:18.088 INFO:teuthology.orchestra.run.smithi067.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-03T17:23:18.088 INFO:teuthology.orchestra.run.smithi067.stdout:mds.all 12/12 39s ago 3m smithi067=a;smithi067=d;smithi067=g;smithi067=j;smithi070=b;smithi070=e;smithi070=h;smithi070=k;smithi082=c;smithi082=f;smithi082=i;smithi082=l;count:12 2024-05-03T17:23:18.088 INFO:teuthology.orchestra.run.smithi067.stdout:mgr 3/3 39s ago 9m smithi067=x;smithi070=y;smithi082=z;count:3 2024-05-03T17:23:18.088 INFO:teuthology.orchestra.run.smithi067.stdout:mon 3/3 39s ago 9m smithi067:172.21.15.67=a;smithi070:172.21.15.70=b;smithi082:172.21.15.82=c;count:3 2024-05-03T17:23:18.088 INFO:teuthology.orchestra.run.smithi067.stdout:osd 12 39s ago - 2024-05-03T17:23:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.089+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/380330956 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe19c0b2910 msgr2=0x7fe19c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.089+0000 7fe1b9a3a700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe19c0b2910 0x7fe19c0b4dd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8005950 tx=0x7fe1a80058e0 comp rx=0 tx=0).stop 2024-05-03T17:23:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.089+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/380330956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b4103550 msgr2=0x7fe1b4199610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.089+0000 7fe1b9a3a700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b4103550 0x7fe1b4199610 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009f50 tx=0x7fe1a40047b0 comp rx=0 tx=0).stop 2024-05-03T17:23:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.089+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/380330956 shutdown_connections 2024-05-03T17:23:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.089+0000 7fe1b9a3a700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe19c0b2910 0x7fe19c0b4dd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.089+0000 7fe1b9a3a700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe1b41050f0 0x7fe1b41a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.089+0000 7fe1b9a3a700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe1b4103550 0x7fe1b4199610 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.089+0000 7fe1b9a3a700 1 --2- 172.21.15.67:0/380330956 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe1b4104750 0x7fe1b4199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:18.090 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.089+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/380330956 >> 172.21.15.67:0/380330956 conn(0x7fe1b40feaf0 msgr2=0x7fe1b4107a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:18.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.089+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/380330956 shutdown_connections 2024-05-03T17:23:18.091 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:18.089+0000 7fe1b9a3a700 1 -- 172.21.15.67:0/380330956 wait complete. 2024-05-03T17:23:18.796 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:18 smithi067 ceph-mon[29731]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:18.824 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-03T17:23:19.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:18 smithi082 ceph-mon[33535]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:19.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:18 smithi070 ceph-mon[33250]: from='client.14991 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:19.072 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:19.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:19 smithi067 ceph-mon[29731]: pgmap v377: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:20.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:19 smithi082 ceph-mon[33535]: pgmap v377: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:20.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:19 smithi070 ceph-mon[33250]: pgmap v377: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:20.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.925+0000 7f4e31994700 1 -- 172.21.15.67:0/1594333768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c103620 msgr2=0x7f4e2c103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:20.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.925+0000 7f4e31994700 1 --2- 172.21.15.67:0/1594333768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c103620 0x7f4e2c103a80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4e140099c0 tx=0x7f4e14009cd0 comp rx=0 tx=0).stop 2024-05-03T17:23:20.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.926+0000 7f4e31994700 1 -- 172.21.15.67:0/1594333768 shutdown_connections 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.926+0000 7f4e31994700 1 --2- 172.21.15.67:0/1594333768 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e2c103fc0 0x7f4e2c10a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.926+0000 7f4e31994700 1 --2- 172.21.15.67:0/1594333768 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e2c102420 0x7f4e2c102840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.926+0000 7f4e31994700 1 --2- 172.21.15.67:0/1594333768 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c103620 0x7f4e2c103a80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.926+0000 7f4e31994700 1 -- 172.21.15.67:0/1594333768 >> 172.21.15.67:0/1594333768 conn(0x7f4e2c0fda00 msgr2=0x7f4e2c0ffe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.926+0000 7f4e31994700 1 -- 172.21.15.67:0/1594333768 shutdown_connections 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.926+0000 7f4e31994700 1 -- 172.21.15.67:0/1594333768 wait complete. 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.926+0000 7f4e31994700 1 Processor -- start 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.926+0000 7f4e31994700 1 -- start start 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.926+0000 7f4e31994700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c102420 0x7f4e2c199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e31994700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e2c103620 0x7f4e2c199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e31994700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e2c103fc0 0x7f4e2c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e31994700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c10dec0 con 0x7f4e2c102420 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e31994700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c10dd40 con 0x7f4e2c103fc0 2024-05-03T17:23:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e31994700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c10e040 con 0x7f4e2c103620 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e2affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c102420 0x7f4e2c199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e2affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c102420 0x7f4e2c199630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53124/0 (socket says 172.21.15.67:53124) 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e2affd700 1 -- 172.21.15.67:0/1980902116 learned_addr learned my addr 172.21.15.67:0/1980902116 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e2b7fe700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e2c103fc0 0x7f4e2c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e2a7fc700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e2c103620 0x7f4e2c199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e2affd700 1 -- 172.21.15.67:0/1980902116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e2c103620 msgr2=0x7f4e2c199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e2affd700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e2c103620 0x7f4e2c199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e2affd700 1 -- 172.21.15.67:0/1980902116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e2c103fc0 msgr2=0x7f4e2c1a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e2affd700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e2c103fc0 0x7f4e2c1a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e2affd700 1 -- 172.21.15.67:0/1980902116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2c1a1290 con 0x7f4e2c102420 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e2affd700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c102420 0x7f4e2c199630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400add0 tx=0x7f4e2400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.927+0000 7f4e30992700 1 -- 172.21.15.67:0/1980902116 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24017070 con 0x7f4e2c102420 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.928+0000 7f4e30992700 1 -- 172.21.15.67:0/1980902116 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e24004bb0 con 0x7f4e2c102420 2024-05-03T17:23:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.928+0000 7f4e31994700 1 -- 172.21.15.67:0/1980902116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e2c1a1580 con 0x7f4e2c102420 2024-05-03T17:23:20.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.928+0000 7f4e30992700 1 -- 172.21.15.67:0/1980902116 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2400cd70 con 0x7f4e2c102420 2024-05-03T17:23:20.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.928+0000 7f4e31994700 1 -- 172.21.15.67:0/1980902116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e2c1a1a70 con 0x7f4e2c102420 2024-05-03T17:23:20.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.929+0000 7f4e31994700 1 -- 172.21.15.67:0/1980902116 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e2c06a190 con 0x7f4e2c102420 2024-05-03T17:23:20.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.931+0000 7f4e30992700 1 -- 172.21.15.67:0/1980902116 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f4e2401a070 con 0x7f4e2c102420 2024-05-03T17:23:20.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.931+0000 7f4e30992700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4e180b2910 0x7f4e180b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:20.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.932+0000 7f4e2a7fc700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4e180b2910 0x7f4e180b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:20.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.932+0000 7f4e30992700 1 -- 172.21.15.67:0/1980902116 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f4e240e15e0 con 0x7f4e2c102420 2024-05-03T17:23:20.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.932+0000 7f4e2a7fc700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4e180b2910 0x7f4e180b4dd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009990 tx=0x7f4e14003960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:20.935+0000 7f4e30992700 1 -- 172.21.15.67:0/1980902116 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f4e240a9190 con 0x7f4e2c102420 2024-05-03T17:23:21.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.154+0000 7f4e31994700 1 -- 172.21.15.67:0/1980902116 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f4e2c06c320 con 0x7f4e180b2910 2024-05-03T17:23:21.156 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.156+0000 7f4e30992700 1 -- 172.21.15.67:0/1980902116 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+187 (secure 0 0 0) 0x7f4e2c06c320 con 0x7f4e180b2910 2024-05-03T17:23:21.156 INFO:teuthology.orchestra.run.smithi067.stdout:HOST ADDR LABELS STATUS 2024-05-03T17:23:21.156 INFO:teuthology.orchestra.run.smithi067.stdout:smithi067 172.21.15.67 2024-05-03T17:23:21.157 INFO:teuthology.orchestra.run.smithi067.stdout:smithi070 172.21.15.70 2024-05-03T17:23:21.157 INFO:teuthology.orchestra.run.smithi067.stdout:smithi082 172.21.15.82 2024-05-03T17:23:21.157 INFO:teuthology.orchestra.run.smithi067.stdout:3 hosts in cluster 2024-05-03T17:23:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.158+0000 7f4e31994700 1 -- 172.21.15.67:0/1980902116 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4e180b2910 msgr2=0x7f4e180b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.158+0000 7f4e31994700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4e180b2910 0x7f4e180b4dd0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4e14009990 tx=0x7f4e14003960 comp rx=0 tx=0).stop 2024-05-03T17:23:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.158+0000 7f4e31994700 1 -- 172.21.15.67:0/1980902116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c102420 msgr2=0x7f4e2c199630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.158+0000 7f4e31994700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c102420 0x7f4e2c199630 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400add0 tx=0x7f4e2400c330 comp rx=0 tx=0).stop 2024-05-03T17:23:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.158+0000 7f4e31994700 1 -- 172.21.15.67:0/1980902116 shutdown_connections 2024-05-03T17:23:21.159 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.158+0000 7f4e31994700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f4e180b2910 0x7f4e180b4dd0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.158+0000 7f4e31994700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f4e2c103fc0 0x7f4e2c1a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.158+0000 7f4e31994700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4e2c102420 0x7f4e2c199630 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.158+0000 7f4e31994700 1 --2- 172.21.15.67:0/1980902116 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f4e2c103620 0x7f4e2c199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.158+0000 7f4e31994700 1 -- 172.21.15.67:0/1980902116 >> 172.21.15.67:0/1980902116 conn(0x7f4e2c0fda00 msgr2=0x7f4e2c06bc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.159+0000 7f4e31994700 1 -- 172.21.15.67:0/1980902116 shutdown_connections 2024-05-03T17:23:21.160 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:21.159+0000 7f4e31994700 1 -- 172.21.15.67:0/1980902116 wait complete. 2024-05-03T17:23:21.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:21 smithi067 ceph-mon[29731]: pgmap v378: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:21.991 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-03T17:23:22.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:21 smithi082 ceph-mon[33535]: pgmap v378: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:22.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:21 smithi070 ceph-mon[33250]: pgmap v378: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:22.246 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:22.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:22 smithi067 ceph-mon[29731]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:23.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:22 smithi082 ceph-mon[33535]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:23.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:22 smithi070 ceph-mon[33250]: from='client.14997 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:23.912 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:23 smithi067 ceph-mon[29731]: pgmap v379: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:24.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:23 smithi082 ceph-mon[33535]: pgmap v379: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:24.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:23 smithi070 ceph-mon[33250]: pgmap v379: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:24.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.185+0000 7f0176f3e700 1 -- 172.21.15.67:0/1058591883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0170103530 msgr2=0x7f0170103950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:24.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.185+0000 7f0176f3e700 1 --2- 172.21.15.67:0/1058591883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0170103530 0x7f0170103950 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f01640099c0 tx=0x7f0164009cd0 comp rx=0 tx=0).stop 2024-05-03T17:23:24.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.185+0000 7f0176f3e700 1 -- 172.21.15.67:0/1058591883 shutdown_connections 2024-05-03T17:23:24.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.185+0000 7f0176f3e700 1 --2- 172.21.15.67:0/1058591883 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01701050d0 0x7f017010b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:24.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.185+0000 7f0176f3e700 1 --2- 172.21.15.67:0/1058591883 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0170103530 0x7f0170103950 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.185+0000 7f0176f3e700 1 --2- 172.21.15.67:0/1058591883 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0170104730 0x7f0170104b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.185+0000 7f0176f3e700 1 -- 172.21.15.67:0/1058591883 >> 172.21.15.67:0/1058591883 conn(0x7f01700feaf0 msgr2=0x7f0170100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.185+0000 7f0176f3e700 1 -- 172.21.15.67:0/1058591883 shutdown_connections 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.185+0000 7f0176f3e700 1 -- 172.21.15.67:0/1058591883 wait complete. 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f0176f3e700 1 Processor -- start 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f0176f3e700 1 -- start start 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f0176f3e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0170103530 0x7f0170077240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f0176f3e700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0170104730 0x7f0170075890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f0176f3e700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01701050d0 0x7f0170075dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f0176f3e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f017010efb0 con 0x7f0170103530 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f0176f3e700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f017010ee30 con 0x7f01701050d0 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f0176f3e700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f017010f130 con 0x7f0170104730 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f0174cda700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0170103530 0x7f0170077240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:24.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f016ffff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0170104730 0x7f0170075890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f01754db700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01701050d0 0x7f0170075dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f01754db700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01701050d0 0x7f0170075dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.70:3300/0 says I am v2:172.21.15.67:38268/0 (socket says 172.21.15.67:38268) 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f016ffff700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0170104730 0x7f0170075890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.82:3300/0 says I am v2:172.21.15.67:57994/0 (socket says 172.21.15.67:57994) 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.186+0000 7f01754db700 1 -- 172.21.15.67:0/2778049250 learned_addr learned my addr 172.21.15.67:0/2778049250 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.187+0000 7f01754db700 1 -- 172.21.15.67:0/2778049250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0170104730 msgr2=0x7f0170075890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.187+0000 7f01754db700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0170104730 0x7f0170075890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.187+0000 7f01754db700 1 -- 172.21.15.67:0/2778049250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0170103530 msgr2=0x7f0170077240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.187+0000 7f01754db700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0170103530 0x7f0170077240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.187+0000 7f01754db700 1 -- 172.21.15.67:0/2778049250 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0170076680 con 0x7f01701050d0 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.187+0000 7f0174cda700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0170103530 0x7f0170077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.187+0000 7f01754db700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01701050d0 0x7f0170075dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f016800ada0 tx=0x7f016800c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.187+0000 7f016dffb700 1 -- 172.21.15.67:0/2778049250 <== mon.2 v2:172.21.15.70:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168017020 con 0x7f01701050d0 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.187+0000 7f016dffb700 1 -- 172.21.15.67:0/2778049250 <== mon.2 v2:172.21.15.70:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f016800cd80 con 0x7f01701050d0 2024-05-03T17:23:24.190 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.187+0000 7f0176f3e700 1 -- 172.21.15.67:0/2778049250 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01701a37f0 con 0x7f01701050d0 2024-05-03T17:23:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.187+0000 7f016dffb700 1 -- 172.21.15.67:0/2778049250 <== mon.2 v2:172.21.15.70:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0168005260 con 0x7f01701050d0 2024-05-03T17:23:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.188+0000 7f0176f3e700 1 -- 172.21.15.67:0/2778049250 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01701a3cb0 con 0x7f01701050d0 2024-05-03T17:23:24.191 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.188+0000 7f0176f3e700 1 -- 172.21.15.67:0/2778049250 --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f017006a190 con 0x7f01701050d0 2024-05-03T17:23:24.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.190+0000 7f016dffb700 1 -- 172.21.15.67:0/2778049250 <== mon.2 v2:172.21.15.70:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f016800a090 con 0x7f01701050d0 2024-05-03T17:23:24.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.190+0000 7f016dffb700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f01600b2910 0x7f01600b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:24.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.191+0000 7f0174cda700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f01600b2910 0x7f01600b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:24.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.191+0000 7f016dffb700 1 -- 172.21.15.67:0/2778049250 <== mon.2 v2:172.21.15.70:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f01680e2130 con 0x7f01701050d0 2024-05-03T17:23:24.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.191+0000 7f0174cda700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f01600b2910 0x7f01600b4dd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0164006010 tx=0x7f016400bc20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:24.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.195+0000 7f016dffb700 1 -- 172.21.15.67:0/2778049250 <== mon.2 v2:172.21.15.70:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f0168021020 con 0x7f01701050d0 2024-05-03T17:23:24.418 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.417+0000 7f0176f3e700 1 -- 172.21.15.67:0/2778049250 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f01700638e0 con 0x7f01600b2910 2024-05-03T17:23:24.419 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.419+0000 7f016dffb700 1 -- 172.21.15.67:0/2778049250 <== mgr.14152 v2:172.21.15.67:6800/3284262914 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f01700638e0 con 0x7f01600b2910 2024-05-03T17:23:24.419 INFO:teuthology.orchestra.run.smithi067.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-03T17:23:24.419 INFO:teuthology.orchestra.run.smithi067.stdout:smithi067 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53440042400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-05-03T17:23:24.419 INFO:teuthology.orchestra.run.smithi067.stdout:smithi070 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB7513002P480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-05-03T17:23:24.420 INFO:teuthology.orchestra.run.smithi067.stdout:smithi082 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400SD400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-03T17:23:24.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.421+0000 7f0176f3e700 1 -- 172.21.15.67:0/2778049250 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f01600b2910 msgr2=0x7f01600b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.421+0000 7f0176f3e700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f01600b2910 0x7f01600b4dd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0164006010 tx=0x7f016400bc20 comp rx=0 tx=0).stop 2024-05-03T17:23:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.421+0000 7f0176f3e700 1 -- 172.21.15.67:0/2778049250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01701050d0 msgr2=0x7f0170075dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.421+0000 7f0176f3e700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01701050d0 0x7f0170075dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f016800ada0 tx=0x7f016800c330 comp rx=0 tx=0).stop 2024-05-03T17:23:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.421+0000 7f0176f3e700 1 -- 172.21.15.67:0/2778049250 shutdown_connections 2024-05-03T17:23:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.421+0000 7f0176f3e700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f01600b2910 0x7f01600b4dd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.421+0000 7f0176f3e700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f01701050d0 0x7f0170075dd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.421+0000 7f0176f3e700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0170103530 0x7f0170077240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.421+0000 7f0176f3e700 1 --2- 172.21.15.67:0/2778049250 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f0170104730 0x7f0170075890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.421+0000 7f0176f3e700 1 -- 172.21.15.67:0/2778049250 >> 172.21.15.67:0/2778049250 conn(0x7f01700feaf0 msgr2=0x7f0170107ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.421+0000 7f0176f3e700 1 -- 172.21.15.67:0/2778049250 shutdown_connections 2024-05-03T17:23:24.422 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:24.421+0000 7f0176f3e700 1 -- 172.21.15.67:0/2778049250 wait complete. 2024-05-03T17:23:25.136 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-03T17:23:25.145 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi067.front.sepia.ceph.com 2024-05-03T17:23:25.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-03T17:23:25.399 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:25.863 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:25 smithi067 ceph-mon[29731]: from='client.24764 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:25.863 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:25 smithi067 ceph-mon[29731]: pgmap v380: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:26.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:25 smithi082 ceph-mon[33535]: from='client.24764 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:26.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:25 smithi082 ceph-mon[33535]: pgmap v380: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:26.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:25 smithi070 ceph-mon[33250]: from='client.24764 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:26.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:25 smithi070 ceph-mon[33250]: pgmap v380: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.221+0000 7f2e87def700 1 -- 172.21.15.67:0/1278794610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e801047d0 msgr2=0x7f2e80104c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.221+0000 7f2e87def700 1 --2- 172.21.15.67:0/1278794610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e801047d0 0x7f2e80104c30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f2e70009990 tx=0x7f2e70009ca0 comp rx=0 tx=0).stop 2024-05-03T17:23:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.221+0000 7f2e87def700 1 -- 172.21.15.67:0/1278794610 shutdown_connections 2024-05-03T17:23:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.221+0000 7f2e87def700 1 --2- 172.21.15.67:0/1278794610 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e80105170 0x7f2e8010ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.221+0000 7f2e87def700 1 --2- 172.21.15.67:0/1278794610 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e801017f0 0x7f2e80101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.221+0000 7f2e87def700 1 --2- 172.21.15.67:0/1278794610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e801047d0 0x7f2e80104c30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.221+0000 7f2e87def700 1 -- 172.21.15.67:0/1278794610 >> 172.21.15.67:0/1278794610 conn(0x7f2e800fd150 msgr2=0x7f2e800ff590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:27.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.221+0000 7f2e87def700 1 -- 172.21.15.67:0/1278794610 shutdown_connections 2024-05-03T17:23:27.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.221+0000 7f2e87def700 1 -- 172.21.15.67:0/1278794610 wait complete. 2024-05-03T17:23:27.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.222+0000 7f2e87def700 1 Processor -- start 2024-05-03T17:23:27.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.222+0000 7f2e87def700 1 -- start start 2024-05-03T17:23:27.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.222+0000 7f2e87def700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e801017f0 0x7f2e80104130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.222+0000 7f2e87def700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e801047d0 0x7f2e80102780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.222+0000 7f2e87def700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e80105170 0x7f2e80102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.222+0000 7f2e87def700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8010efc0 con 0x7f2e801017f0 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.222+0000 7f2e87def700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8010ee40 con 0x7f2e80105170 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.222+0000 7f2e87def700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f2e8010f140 con 0x7f2e801047d0 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.222+0000 7f2e85b8b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e801017f0 0x7f2e80104130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.222+0000 7f2e85b8b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e801017f0 0x7f2e80104130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37648/0 (socket says 172.21.15.67:37648) 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.222+0000 7f2e85b8b700 1 -- 172.21.15.67:0/1619779572 learned_addr learned my addr 172.21.15.67:0/1619779572 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.222+0000 7f2e8538a700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e801047d0 0x7f2e80102780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.223+0000 7f2e8638c700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e80105170 0x7f2e80102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.223+0000 7f2e85b8b700 1 -- 172.21.15.67:0/1619779572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e801047d0 msgr2=0x7f2e80102780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.223+0000 7f2e85b8b700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e801047d0 0x7f2e80102780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:27.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.223+0000 7f2e85b8b700 1 -- 172.21.15.67:0/1619779572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e80105170 msgr2=0x7f2e80102cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:27.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.223+0000 7f2e85b8b700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e80105170 0x7f2e80102cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:27.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.223+0000 7f2e85b8b700 1 -- 172.21.15.67:0/1619779572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e80103570 con 0x7f2e801017f0 2024-05-03T17:23:27.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.223+0000 7f2e85b8b700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e801017f0 0x7f2e80104130 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c00c990 tx=0x7f2e7c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:27.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.223+0000 7f2e76ffd700 1 -- 172.21.15.67:0/1619779572 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c007b00 con 0x7f2e801017f0 2024-05-03T17:23:27.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.223+0000 7f2e76ffd700 1 -- 172.21.15.67:0/1619779572 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e7c007ca0 con 0x7f2e801017f0 2024-05-03T17:23:27.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.223+0000 7f2e76ffd700 1 -- 172.21.15.67:0/1619779572 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c018660 con 0x7f2e801017f0 2024-05-03T17:23:27.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.223+0000 7f2e87def700 1 -- 172.21.15.67:0/1619779572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e801a3700 con 0x7f2e801017f0 2024-05-03T17:23:27.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.224+0000 7f2e87def700 1 -- 172.21.15.67:0/1619779572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e801a3c40 con 0x7f2e801017f0 2024-05-03T17:23:27.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.224+0000 7f2e87def700 1 -- 172.21.15.67:0/1619779572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e8006a190 con 0x7f2e801017f0 2024-05-03T17:23:27.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.226+0000 7f2e76ffd700 1 -- 172.21.15.67:0/1619779572 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f2e7c004030 con 0x7f2e801017f0 2024-05-03T17:23:27.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.227+0000 7f2e76ffd700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f2e6c0b2990 0x7f2e6c0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:27.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.227+0000 7f2e8538a700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f2e6c0b2990 0x7f2e6c0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:27.227 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.227+0000 7f2e76ffd700 1 -- 172.21.15.67:0/1619779572 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f2e7c0e2380 con 0x7f2e801017f0 2024-05-03T17:23:27.228 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.227+0000 7f2e8538a700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f2e6c0b2990 0x7f2e6c0b4e50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2e7000b5c0 tx=0x7f2e70005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:27.231 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.230+0000 7f2e76ffd700 1 -- 172.21.15.67:0/1619779572 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f2e7c014030 con 0x7f2e801017f0 2024-05-03T17:23:27.496 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.496+0000 7f2e87def700 1 -- 172.21.15.67:0/1619779572 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f2e8002e5d0 con 0x7f2e801017f0 2024-05-03T17:23:27.497 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.496+0000 7f2e76ffd700 1 -- 172.21.15.67:0/1619779572 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 25 v25) ==== 76+0+3036 (secure 0 0 0) 0x7f2e7c0aa040 con 0x7f2e801017f0 2024-05-03T17:23:27.497 INFO:teuthology.orchestra.run.smithi067.stdout:e25 2024-05-03T17:23:27.497 INFO:teuthology.orchestra.run.smithi067.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-03T17:23:27.497 INFO:teuthology.orchestra.run.smithi067.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-03T17:23:27.497 INFO:teuthology.orchestra.run.smithi067.stdout:legacy client fscid: 1 2024-05-03T17:23:27.497 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:23:27.497 INFO:teuthology.orchestra.run.smithi067.stdout:Filesystem 'cephfs' (1) 2024-05-03T17:23:27.497 INFO:teuthology.orchestra.run.smithi067.stdout:fs_name cephfs 2024-05-03T17:23:27.497 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 25 2024-05-03T17:23:27.497 INFO:teuthology.orchestra.run.smithi067.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-03T17:23:27.497 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-05-03T17:20:39.322107+0000 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-05-03T17:21:01.753515+0000 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:tableserver 0 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:root 0 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:session_timeout 300 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:session_autoclose 300 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:max_file_size 1099511627776 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:max_xattr_size 65536 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:required_client_features {} 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure 0 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:last_failure_osd_epoch 0 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:max_mds 3 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:in 0,1,2 2024-05-03T17:23:27.498 INFO:teuthology.orchestra.run.smithi067.stdout:up {0=24464,1=24446,2=24442} 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:failed 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:damaged 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:stopped 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:data_pools [3] 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:metadata_pool 2 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:inline_data disabled 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:balancer 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:bal_rank_mask -1 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:standby_count_wanted 0 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:qdb_cluster leader: 24464 members: 24446,24464,24442 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b{0:24464} state up:active seq 6 addr [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c{0:24427} state up:standby-replay seq 1 addr [v2:172.21.15.82:6836/370282857,v1:172.21.15.82:6837/370282857] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.h{1:24446} state up:active seq 13 addr [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.e{1:24425} state up:standby-replay seq 1 addr [v2:172.21.15.70:6834/1966549782,v1:172.21.15.70:6835/1966549782] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.i{2:24442} state up:active seq 12 addr [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T17:23:27.499 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.f{2:24421} state up:standby-replay seq 1 addr [v2:172.21.15.82:6834/1568454842,v1:172.21.15.82:6835/1568454842] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T17:23:27.500 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:23:27.500 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:23:27.500 INFO:teuthology.orchestra.run.smithi067.stdout:Standby daemons: 2024-05-03T17:23:27.500 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:23:27.500 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.j{-1:14505} state up:standby seq 1 addr [v2:172.21.15.67:6834/1814275265,v1:172.21.15.67:6835/1814275265] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T17:23:27.500 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d{-1:14517} state up:standby seq 1 addr [v2:172.21.15.67:6836/583931110,v1:172.21.15.67:6837/583931110] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T17:23:27.500 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.g{-1:14565} state up:standby seq 1 addr [v2:172.21.15.67:6838/316463706,v1:172.21.15.67:6839/316463706] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T17:23:27.500 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a{-1:14577} state up:standby seq 1 addr [v2:172.21.15.67:6840/3422086111,v1:172.21.15.67:6841/3422086111] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T17:23:27.500 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.l{-1:24394} state up:standby seq 1 addr [v2:172.21.15.82:6832/3927012974,v1:172.21.15.82:6833/3927012974] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T17:23:27.500 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.k{-1:24398} state up:standby seq 1 addr [v2:172.21.15.70:6832/2034294041,v1:172.21.15.70:6833/2034294041] compat {c=[1],r=[1],i=[1fff]}] 2024-05-03T17:23:27.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.499+0000 7f2e87def700 1 -- 172.21.15.67:0/1619779572 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f2e6c0b2990 msgr2=0x7f2e6c0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:27.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.499+0000 7f2e87def700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f2e6c0b2990 0x7f2e6c0b4e50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f2e7000b5c0 tx=0x7f2e70005fb0 comp rx=0 tx=0).stop 2024-05-03T17:23:27.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.499+0000 7f2e87def700 1 -- 172.21.15.67:0/1619779572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e801017f0 msgr2=0x7f2e80104130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:27.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.499+0000 7f2e87def700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e801017f0 0x7f2e80104130 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c00c990 tx=0x7f2e7c00cca0 comp rx=0 tx=0).stop 2024-05-03T17:23:27.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.499+0000 7f2e87def700 1 -- 172.21.15.67:0/1619779572 shutdown_connections 2024-05-03T17:23:27.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.499+0000 7f2e87def700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f2e6c0b2990 0x7f2e6c0b4e50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:27.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.499+0000 7f2e87def700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f2e80105170 0x7f2e80102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:27.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.499+0000 7f2e87def700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2e801017f0 0x7f2e80104130 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:27.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.499+0000 7f2e87def700 1 --2- 172.21.15.67:0/1619779572 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f2e801047d0 0x7f2e80102780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:27.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.499+0000 7f2e87def700 1 -- 172.21.15.67:0/1619779572 >> 172.21.15.67:0/1619779572 conn(0x7f2e800fd150 msgr2=0x7f2e80107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:27.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.500+0000 7f2e87def700 1 -- 172.21.15.67:0/1619779572 shutdown_connections 2024-05-03T17:23:27.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:27.500+0000 7f2e87def700 1 -- 172.21.15.67:0/1619779572 wait complete. 2024-05-03T17:23:27.502 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T17:23:27.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:27 smithi067 ceph-mon[29731]: pgmap v381: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:27.876 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:27 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1619779572' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-03T17:23:28.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:27 smithi082 ceph-mon[33535]: pgmap v381: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:28.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:27 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1619779572' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-03T17:23:28.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:27 smithi070 ceph-mon[33250]: pgmap v381: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:28.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:27 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1619779572' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-03T17:23:28.221 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-03T17:23:28.471 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:29.891 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:29 smithi067 ceph-mon[29731]: pgmap v382: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:30.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:29 smithi082 ceph-mon[33535]: pgmap v382: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:30.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:29 smithi070 ceph-mon[33250]: pgmap v382: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.280+0000 7fe50ddeb700 1 -- 172.21.15.67:0/2971089655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe508104720 msgr2=0x7fe508104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.280+0000 7fe50ddeb700 1 --2- 172.21.15.67:0/2971089655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe508104720 0x7fe508104b80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc0099c0 tx=0x7fe4fc009cd0 comp rx=0 tx=0).stop 2024-05-03T17:23:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.281+0000 7fe50ddeb700 1 -- 172.21.15.67:0/2971089655 shutdown_connections 2024-05-03T17:23:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.281+0000 7fe50ddeb700 1 --2- 172.21.15.67:0/2971089655 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5081050c0 0x7fe50810b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.281+0000 7fe50ddeb700 1 --2- 172.21.15.67:0/2971089655 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe508103520 0x7fe508103940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:30.283 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.281+0000 7fe50ddeb700 1 --2- 172.21.15.67:0/2971089655 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe508104720 0x7fe508104b80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.281+0000 7fe50ddeb700 1 -- 172.21.15.67:0/2971089655 >> 172.21.15.67:0/2971089655 conn(0x7fe5080feac0 msgr2=0x7fe508100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.281+0000 7fe50ddeb700 1 -- 172.21.15.67:0/2971089655 shutdown_connections 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.281+0000 7fe50ddeb700 1 -- 172.21.15.67:0/2971089655 wait complete. 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.281+0000 7fe50ddeb700 1 Processor -- start 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.281+0000 7fe50ddeb700 1 -- start start 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe50ddeb700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe508103520 0x7fe508199610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe50ddeb700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe508104720 0x7fe508199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe50ddeb700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5081050c0 0x7fe5081a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe50ddeb700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe50810efa0 con 0x7fe508104720 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe50ddeb700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe50810ee20 con 0x7fe5081050c0 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe50ddeb700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe50810f120 con 0x7fe508103520 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe506ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe508104720 0x7fe508199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe506ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe508104720 0x7fe508199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37672/0 (socket says 172.21.15.67:37672) 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe506ffd700 1 -- 172.21.15.67:0/3752443915 learned_addr learned my addr 172.21.15.67:0/3752443915 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:30.284 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe5077fe700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe508103520 0x7fe508199610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe507fff700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5081050c0 0x7fe5081a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe506ffd700 1 -- 172.21.15.67:0/3752443915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe508103520 msgr2=0x7fe508199610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe506ffd700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe508103520 0x7fe508199610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe506ffd700 1 -- 172.21.15.67:0/3752443915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5081050c0 msgr2=0x7fe5081a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe506ffd700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5081050c0 0x7fe5081a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe506ffd700 1 -- 172.21.15.67:0/3752443915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5081a1270 con 0x7fe508104720 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe5077fe700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe508103520 0x7fe508199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe507fff700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5081050c0 0x7fe5081a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe506ffd700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe508104720 0x7fe508199b50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc005910 tx=0x7fe4fc005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe504ff9700 1 -- 172.21.15.67:0/3752443915 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc027070 con 0x7fe508104720 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe50ddeb700 1 -- 172.21.15.67:0/3752443915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5081a1500 con 0x7fe508104720 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe504ff9700 1 -- 172.21.15.67:0/3752443915 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe4fc022470 con 0x7fe508104720 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.282+0000 7fe504ff9700 1 -- 172.21.15.67:0/3752443915 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4fc00f6c0 con 0x7fe508104720 2024-05-03T17:23:30.285 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.283+0000 7fe50ddeb700 1 -- 172.21.15.67:0/3752443915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5081a19e0 con 0x7fe508104720 2024-05-03T17:23:30.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.284+0000 7fe50ddeb700 1 -- 172.21.15.67:0/3752443915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe50806a190 con 0x7fe508104720 2024-05-03T17:23:30.286 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.286+0000 7fe504ff9700 1 -- 172.21.15.67:0/3752443915 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fe4fc02a070 con 0x7fe508104720 2024-05-03T17:23:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.286+0000 7fe504ff9700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe4f00b2910 0x7fe4f00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.286+0000 7fe5077fe700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe4f00b2910 0x7fe4f00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.287+0000 7fe504ff9700 1 -- 172.21.15.67:0/3752443915 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fe4fc0e3700 con 0x7fe508104720 2024-05-03T17:23:30.287 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.287+0000 7fe5077fe700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe4f00b2910 0x7fe4f00b4dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8005950 tx=0x7fe4f80058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:30.290 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.290+0000 7fe504ff9700 1 -- 172.21.15.67:0/3752443915 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe4fc031080 con 0x7fe508104720 2024-05-03T17:23:30.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.504+0000 7fe50ddeb700 1 -- 172.21.15.67:0/3752443915 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fe50802e510 con 0x7fe508104720 2024-05-03T17:23:30.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.505+0000 7fe504ff9700 1 -- 172.21.15.67:0/3752443915 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v81) ==== 56+0+5287 (secure 0 0 0) 0x7fe4fc013070 con 0x7fe508104720 2024-05-03T17:23:30.505 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 81 2024-05-03T17:23:30.505 INFO:teuthology.orchestra.run.smithi067.stdout:fsid 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:23:30.505 INFO:teuthology.orchestra.run.smithi067.stdout:created 2024-05-03T17:12:11.709381+0000 2024-05-03T17:23:30.505 INFO:teuthology.orchestra.run.smithi067.stdout:modified 2024-05-03T17:21:07.584731+0000 2024-05-03T17:23:30.505 INFO:teuthology.orchestra.run.smithi067.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:crush_version 26 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:full_ratio 0.95 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:backfillfull_ratio 0.9 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:nearfull_ratio 0.85 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:require_min_compat_client luminous 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:min_compat_client luminous 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:require_osd_release squid 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:stretch_mode_enabled false 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:max_osd 12 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:osd.0 up in weight 1 up_from 8 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6802/1754365741,v1:172.21.15.67:6803/1754365741] [v2:172.21.15.67:6804/1754365741,v1:172.21.15.67:6805/1754365741] exists,up 80962eb1-a506-49bb-a528-dee035eb4322 2024-05-03T17:23:30.506 INFO:teuthology.orchestra.run.smithi067.stdout:osd.1 up in weight 1 up_from 13 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6800/1006339176,v1:172.21.15.70:6801/1006339176] [v2:172.21.15.70:6802/1006339176,v1:172.21.15.70:6803/1006339176] exists,up d7721d5f-4373-4cb2-bb57-bc368d6bb818 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:osd.2 up in weight 1 up_from 18 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6800/1888803264,v1:172.21.15.82:6801/1888803264] [v2:172.21.15.82:6802/1888803264,v1:172.21.15.82:6803/1888803264] exists,up 9992fd98-821b-40ce-b051-89befbfd87c8 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:osd.3 up in weight 1 up_from 26 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6810/313717723,v1:172.21.15.67:6811/313717723] [v2:172.21.15.67:6812/313717723,v1:172.21.15.67:6813/313717723] exists,up 14bfe6a6-0bc8-4766-8d36-fbaecc61a396 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:osd.4 up in weight 1 up_from 32 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6808/3526300828,v1:172.21.15.70:6809/3526300828] [v2:172.21.15.70:6810/3526300828,v1:172.21.15.70:6811/3526300828] exists,up 5a44aeb8-aabf-4824-be65-2c7170dbee52 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:osd.5 up in weight 1 up_from 37 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6808/3754223290,v1:172.21.15.82:6809/3754223290] [v2:172.21.15.82:6810/3754223290,v1:172.21.15.82:6811/3754223290] exists,up a6ac56df-466e-4096-8fbf-835f5b544297 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:osd.6 up in weight 1 up_from 43 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6818/4033769161,v1:172.21.15.67:6819/4033769161] [v2:172.21.15.67:6820/4033769161,v1:172.21.15.67:6821/4033769161] exists,up 4d8a5e8a-66ac-462e-b960-24c0ee248fac 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:osd.7 up in weight 1 up_from 48 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6816/3366146173,v1:172.21.15.70:6817/3366146173] [v2:172.21.15.70:6818/3366146173,v1:172.21.15.70:6819/3366146173] exists,up eb983837-8172-4802-9843-6360cda8f4be 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:osd.8 up in weight 1 up_from 54 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6816/347699286,v1:172.21.15.82:6817/347699286] [v2:172.21.15.82:6818/347699286,v1:172.21.15.82:6819/347699286] exists,up c9d4ce56-e940-48d1-a9b5-437139e3fed0 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:osd.9 up in weight 1 up_from 60 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.67:6826/2600095198,v1:172.21.15.67:6827/2600095198] [v2:172.21.15.67:6828/2600095198,v1:172.21.15.67:6829/2600095198] exists,up 09596d19-f2d7-485e-a028-9ce5468c6651 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:osd.10 up in weight 1 up_from 65 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.70:6824/2897583629,v1:172.21.15.70:6825/2897583629] [v2:172.21.15.70:6826/2897583629,v1:172.21.15.70:6827/2897583629] exists,up ec4cffbc-0127-47e5-8176-0972f74a5514 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:osd.11 up in weight 1 up_from 71 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.82:6824/4082656149,v1:172.21.15.82:6825/4082656149] [v2:172.21.15.82:6826/4082656149,v1:172.21.15.82:6827/4082656149] exists,up a8acfb0b-98e8-49e1-a114-f33aa8160b62 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 2.5 [8,11] 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 2.b [8,11,2,1] 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 2.16 [2,4] 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.17 [4,8] 2024-05-03T17:23:30.507 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.19 [3,7] 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.1d [5,9] 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.24 [4,9] 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.33 [5,9] 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:pg_upmap_items 3.3a [4,6] 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/2131289004 expires 2024-05-04T17:13:06.324002+0000 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:6801/3335149382 expires 2024-05-04T17:13:06.324002+0000 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/992151181 expires 2024-05-04T17:13:06.324002+0000 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:6801/1345903711 expires 2024-05-04T17:12:38.899429+0000 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:6800/1345903711 expires 2024-05-04T17:12:38.899429+0000 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/2614686067 expires 2024-05-04T17:12:38.899429+0000 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/3137329076 expires 2024-05-04T17:12:38.899429+0000 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/2323686292 expires 2024-05-04T17:13:06.324002+0000 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:0/2579128777 expires 2024-05-04T17:12:38.899429+0000 2024-05-03T17:23:30.508 INFO:teuthology.orchestra.run.smithi067.stdout:blocklist 172.21.15.67:6800/3335149382 expires 2024-05-04T17:13:06.324002+0000 2024-05-03T17:23:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.507+0000 7fe50ddeb700 1 -- 172.21.15.67:0/3752443915 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe4f00b2910 msgr2=0x7fe4f00b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.507+0000 7fe50ddeb700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe4f00b2910 0x7fe4f00b4dd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe4f8005950 tx=0x7fe4f80058e0 comp rx=0 tx=0).stop 2024-05-03T17:23:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.507+0000 7fe50ddeb700 1 -- 172.21.15.67:0/3752443915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe508104720 msgr2=0x7fe508199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.507+0000 7fe50ddeb700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe508104720 0x7fe508199b50 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe4fc005910 tx=0x7fe4fc005e80 comp rx=0 tx=0).stop 2024-05-03T17:23:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.507+0000 7fe50ddeb700 1 -- 172.21.15.67:0/3752443915 shutdown_connections 2024-05-03T17:23:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.507+0000 7fe50ddeb700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe4f00b2910 0x7fe4f00b4dd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.507+0000 7fe50ddeb700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe5081050c0 0x7fe5081a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:30.509 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.507+0000 7fe50ddeb700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe508103520 0x7fe508199610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:30.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.507+0000 7fe50ddeb700 1 --2- 172.21.15.67:0/3752443915 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe508104720 0x7fe508199b50 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:30.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.507+0000 7fe50ddeb700 1 -- 172.21.15.67:0/3752443915 >> 172.21.15.67:0/3752443915 conn(0x7fe5080feac0 msgr2=0x7fe508107aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:30.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.507+0000 7fe50ddeb700 1 -- 172.21.15.67:0/3752443915 shutdown_connections 2024-05-03T17:23:30.510 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:30.507+0000 7fe50ddeb700 1 -- 172.21.15.67:0/3752443915 wait complete. 2024-05-03T17:23:30.877 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:30 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3752443915' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-03T17:23:31.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:30 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3752443915' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-03T17:23:31.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:30 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3752443915' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-03T17:23:31.215 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-03T17:23:31.227 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:23:31.477 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:31.949 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:31 smithi067 ceph-mon[29731]: pgmap v383: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:32.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:31 smithi082 ceph-mon[33535]: pgmap v383: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:31 smithi070 ceph-mon[33250]: pgmap v383: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.247+0000 7fe8facb1700 1 -- 172.21.15.67:0/783466730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f4102d60 msgr2=0x7fe8f410f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.247+0000 7fe8facb1700 1 --2- 172.21.15.67:0/783466730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f4102d60 0x7fe8f410f250 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4009990 tx=0x7fe8e4009ca0 comp rx=0 tx=0).stop 2024-05-03T17:23:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.247+0000 7fe8facb1700 1 -- 172.21.15.67:0/783466730 shutdown_connections 2024-05-03T17:23:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.247+0000 7fe8facb1700 1 --2- 172.21.15.67:0/783466730 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f4102d60 0x7fe8f410f250 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.247+0000 7fe8facb1700 1 --2- 172.21.15.67:0/783466730 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8f4108410 0x7fe8f41087f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.247+0000 7fe8facb1700 1 --2- 172.21.15.67:0/783466730 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8f41023a0 0x7fe8f4102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.247+0000 7fe8facb1700 1 -- 172.21.15.67:0/783466730 >> 172.21.15.67:0/783466730 conn(0x7fe8f4079350 msgr2=0x7fe8f4079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.247+0000 7fe8facb1700 1 -- 172.21.15.67:0/783466730 shutdown_connections 2024-05-03T17:23:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.247+0000 7fe8facb1700 1 -- 172.21.15.67:0/783466730 wait complete. 2024-05-03T17:23:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8facb1700 1 Processor -- start 2024-05-03T17:23:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8facb1700 1 -- start start 2024-05-03T17:23:33.250 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8facb1700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8f41023a0 0x7fe8f41a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8facb1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f4102d60 0x7fe8f41a1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8facb1700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8f4108410 0x7fe8f419bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8facb1700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f4112530 con 0x7fe8f4102d60 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8facb1700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f41123b0 con 0x7fe8f4108410 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8facb1700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fe8f41126b0 con 0x7fe8f41023a0 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8f3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f4102d60 0x7fe8f41a1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8f3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f4102d60 0x7fe8f41a1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37696/0 (socket says 172.21.15.67:37696) 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8f3fff700 1 -- 172.21.15.67:0/3043214479 learned_addr learned my addr 172.21.15.67:0/3043214479 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8f8a4d700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8f41023a0 0x7fe8f41a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8f924e700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8f4108410 0x7fe8f419bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8f3fff700 1 -- 172.21.15.67:0/3043214479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8f41023a0 msgr2=0x7fe8f41a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8f3fff700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8f41023a0 0x7fe8f41a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8f3fff700 1 -- 172.21.15.67:0/3043214479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8f4108410 msgr2=0x7fe8f419bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:33.251 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8f3fff700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8f4108410 0x7fe8f419bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8f3fff700 1 -- 172.21.15.67:0/3043214479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8f419c3f0 con 0x7fe8f4102d60 2024-05-03T17:23:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.248+0000 7fe8f924e700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8f4108410 0x7fe8f419bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:23:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.249+0000 7fe8f3fff700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f4102d60 0x7fe8f41a1f80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800e9a0 tx=0x7fe8e800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.249+0000 7fe8f1ffb700 1 -- 172.21.15.67:0/3043214479 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e800cbd0 con 0x7fe8f4102d60 2024-05-03T17:23:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.249+0000 7fe8f1ffb700 1 -- 172.21.15.67:0/3043214479 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8e800cd70 con 0x7fe8f4102d60 2024-05-03T17:23:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.249+0000 7fe8facb1700 1 -- 172.21.15.67:0/3043214479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8f419c6e0 con 0x7fe8f4102d60 2024-05-03T17:23:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.249+0000 7fe8f1ffb700 1 -- 172.21.15.67:0/3043214479 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8e8018680 con 0x7fe8f4102d60 2024-05-03T17:23:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.249+0000 7fe8facb1700 1 -- 172.21.15.67:0/3043214479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8f41a89c0 con 0x7fe8f4102d60 2024-05-03T17:23:33.252 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.250+0000 7fe8facb1700 1 -- 172.21.15.67:0/3043214479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8f4103ac0 con 0x7fe8f4102d60 2024-05-03T17:23:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.252+0000 7fe8f1ffb700 1 -- 172.21.15.67:0/3043214479 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fe8e8010430 con 0x7fe8f4102d60 2024-05-03T17:23:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.253+0000 7fe8f1ffb700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe8dc0b2990 0x7fe8dc0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.253+0000 7fe8f8a4d700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe8dc0b2990 0x7fe8dc0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:33.253 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.253+0000 7fe8f1ffb700 1 -- 172.21.15.67:0/3043214479 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fe8e80e2a60 con 0x7fe8f4102d60 2024-05-03T17:23:33.254 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.253+0000 7fe8f8a4d700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe8dc0b2990 0x7fe8dc0b4e50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00f540 tx=0x7fe8ec00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:33.257 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.257+0000 7fe8f1ffb700 1 -- 172.21.15.67:0/3043214479 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fe8e80aa610 con 0x7fe8f4102d60 2024-05-03T17:23:33.523 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.523+0000 7fe8facb1700 1 -- 172.21.15.67:0/3043214479 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe8f41a8b90 con 0x7fe8f4102d60 2024-05-03T17:23:33.524 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.523+0000 7fe8f1ffb700 1 -- 172.21.15.67:0/3043214479 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11850 (secure 0 0 0) 0x7fe8e80a9f10 con 0x7fe8f4102d60 2024-05-03T17:23:33.524 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:23:33.524 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/316463706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":316463706},{"type":"v1","addr":"172.21.15.67:6839","nonce":316463706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3422086111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3422086111},{"type":"v1","addr":"172.21.15.67:6841","nonce":3422086111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T17:20:39.322107+0000","modified":"2024-05-03T17:21:01.753515+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24464,"mds_1":24446,"mds_2":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24421":{"gid":24421,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1568454842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1568454842},{"type":"v1","addr":"172.21.15.82:6835","nonce":1568454842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/370282857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":370282857},{"type":"v1","addr":"172.21.15.82:6837","nonce":370282857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.82:6839/2084218047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2084218047},{"type":"v1","addr":"172.21.15.82:6839","nonce":2084218047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.70:6837/3323713018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3323713018},{"type":"v1","addr":"172.21.15.70:6837","nonce":3323713018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6839/902276547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":902276547},{"type":"v1","addr":"172.21.15.70:6839","nonce":902276547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24446,24464,24442]},"id":1}]} 2024-05-03T17:23:33.526 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.526+0000 7fe8facb1700 1 -- 172.21.15.67:0/3043214479 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe8dc0b2990 msgr2=0x7fe8dc0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.526+0000 7fe8facb1700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe8dc0b2990 0x7fe8dc0b4e50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00f540 tx=0x7fe8ec00b040 comp rx=0 tx=0).stop 2024-05-03T17:23:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.526+0000 7fe8facb1700 1 -- 172.21.15.67:0/3043214479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f4102d60 msgr2=0x7fe8f41a1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.526+0000 7fe8facb1700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f4102d60 0x7fe8f41a1f80 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe8e800e9a0 tx=0x7fe8e800ecb0 comp rx=0 tx=0).stop 2024-05-03T17:23:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.526+0000 7fe8facb1700 1 -- 172.21.15.67:0/3043214479 shutdown_connections 2024-05-03T17:23:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.526+0000 7fe8facb1700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fe8dc0b2990 0x7fe8dc0b4e50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.526+0000 7fe8facb1700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fe8f4108410 0x7fe8f419bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.526+0000 7fe8facb1700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fe8f41023a0 0x7fe8f41a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.526+0000 7fe8facb1700 1 --2- 172.21.15.67:0/3043214479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fe8f4102d60 0x7fe8f41a1f80 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.526+0000 7fe8facb1700 1 -- 172.21.15.67:0/3043214479 >> 172.21.15.67:0/3043214479 conn(0x7fe8f4079350 msgr2=0x7fe8f41014a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.526+0000 7fe8facb1700 1 -- 172.21.15.67:0/3043214479 shutdown_connections 2024-05-03T17:23:33.527 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:33.526+0000 7fe8facb1700 1 -- 172.21.15.67:0/3043214479 wait complete. 2024-05-03T17:23:33.528 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T17:23:33.895 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:33 smithi067 ceph-mon[29731]: pgmap v384: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:33.895 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:33 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3043214479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:23:34.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:33 smithi082 ceph-mon[33535]: pgmap v384: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:34.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:33 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3043214479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:23:34.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:33 smithi070 ceph-mon[33250]: pgmap v384: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:34.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:33 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3043214479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:23:34.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:23:34.604 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:36.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:35 smithi082 ceph-mon[33535]: pgmap v385: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:36.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:35 smithi070 ceph-mon[33250]: pgmap v385: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:36.096 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:35 smithi067 ceph-mon[29731]: pgmap v385: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:36.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.498+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1858252077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e8109530 msgr2=0x7fb5e8109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:36.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.498+0000 7fb5ed1fe700 1 --2- 172.21.15.67:0/1858252077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e8109530 0x7fb5e8109910 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8009990 tx=0x7fb5d8009ca0 comp rx=0 tx=0).stop 2024-05-03T17:23:36.501 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.498+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1858252077 shutdown_connections 2024-05-03T17:23:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.498+0000 7fb5ed1fe700 1 --2- 172.21.15.67:0/1858252077 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5e806c590 0x7fb5e8110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.498+0000 7fb5ed1fe700 1 --2- 172.21.15.67:0/1858252077 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e8109530 0x7fb5e8109910 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.498+0000 7fb5ed1fe700 1 --2- 172.21.15.67:0/1858252077 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5e806bbd0 0x7fb5e806c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.498+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1858252077 >> 172.21.15.67:0/1858252077 conn(0x7fb5e8079340 msgr2=0x7fb5e8079750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.498+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1858252077 shutdown_connections 2024-05-03T17:23:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.498+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1858252077 wait complete. 2024-05-03T17:23:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.499+0000 7fb5ed1fe700 1 Processor -- start 2024-05-03T17:23:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.499+0000 7fb5ed1fe700 1 -- start start 2024-05-03T17:23:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.499+0000 7fb5ed1fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e806bbd0 0x7fb5e81024a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.499+0000 7fb5ed1fe700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5e806c590 0x7fb5e81029e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.499+0000 7fb5ed1fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5e8109530 0x7fb5e8102f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:36.502 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.499+0000 7fb5ed1fe700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e8113690 con 0x7fb5e806bbd0 2024-05-03T17:23:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5ed1fe700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e8113510 con 0x7fb5e806c590 2024-05-03T17:23:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5ed1fe700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e8113810 con 0x7fb5e8109530 2024-05-03T17:23:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e6d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e806bbd0 0x7fb5e81024a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e6d9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e806bbd0 0x7fb5e81024a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:37716/0 (socket says 172.21.15.67:37716) 2024-05-03T17:23:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e6d9d700 1 -- 172.21.15.67:0/1728985200 learned_addr learned my addr 172.21.15.67:0/1728985200 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e759e700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5e8109530 0x7fb5e8102f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5dffff700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5e806c590 0x7fb5e81029e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e6d9d700 1 -- 172.21.15.67:0/1728985200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5e8109530 msgr2=0x7fb5e8102f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e6d9d700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5e8109530 0x7fb5e8102f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e6d9d700 1 -- 172.21.15.67:0/1728985200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5e806c590 msgr2=0x7fb5e81029e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e6d9d700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5e806c590 0x7fb5e81029e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:36.503 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e6d9d700 1 -- 172.21.15.67:0/1728985200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e81a8110 con 0x7fb5e806bbd0 2024-05-03T17:23:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5dffff700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5e806c590 0x7fb5e81029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:23:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e6d9d700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e806bbd0 0x7fb5e81024a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8003ac0 tx=0x7fb5d8003af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e4d99700 1 -- 172.21.15.67:0/1728985200 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8027070 con 0x7fb5e806bbd0 2024-05-03T17:23:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1728985200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5e81a83a0 con 0x7fb5e806bbd0 2024-05-03T17:23:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e4d99700 1 -- 172.21.15.67:0/1728985200 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb5d8003680 con 0x7fb5e806bbd0 2024-05-03T17:23:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5e4d99700 1 -- 172.21.15.67:0/1728985200 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8022c10 con 0x7fb5e806bbd0 2024-05-03T17:23:36.504 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.500+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1728985200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5e81a88d0 con 0x7fb5e806bbd0 2024-05-03T17:23:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.501+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1728985200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5e8104bd0 con 0x7fb5e806bbd0 2024-05-03T17:23:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.503+0000 7fb5e4d99700 1 -- 172.21.15.67:0/1728985200 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fb5d802a070 con 0x7fb5e806bbd0 2024-05-03T17:23:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.503+0000 7fb5e4d99700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb5d40b2910 0x7fb5d40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.503+0000 7fb5dffff700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb5d40b2910 0x7fb5d40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.503+0000 7fb5e4d99700 1 -- 172.21.15.67:0/1728985200 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fb5d80e32c0 con 0x7fb5e806bbd0 2024-05-03T17:23:36.505 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.504+0000 7fb5dffff700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb5d40b2910 0x7fb5d40b4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb5e80799b0 tx=0x7fb5d0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:36.508 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.508+0000 7fb5e4d99700 1 -- 172.21.15.67:0/1728985200 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fb5d8013070 con 0x7fb5e806bbd0 2024-05-03T17:23:36.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.779+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1728985200 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb5e81039d0 con 0x7fb5e806bbd0 2024-05-03T17:23:36.780 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.780+0000 7fb5e4d99700 1 -- 172.21.15.67:0/1728985200 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11850 (secure 0 0 0) 0x7fb5d80aaf00 con 0x7fb5e806bbd0 2024-05-03T17:23:36.781 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:23:36.781 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/316463706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":316463706},{"type":"v1","addr":"172.21.15.67:6839","nonce":316463706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3422086111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3422086111},{"type":"v1","addr":"172.21.15.67:6841","nonce":3422086111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T17:20:39.322107+0000","modified":"2024-05-03T17:21:01.753515+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24464,"mds_1":24446,"mds_2":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24421":{"gid":24421,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1568454842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1568454842},{"type":"v1","addr":"172.21.15.82:6835","nonce":1568454842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/370282857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":370282857},{"type":"v1","addr":"172.21.15.82:6837","nonce":370282857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.82:6839/2084218047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2084218047},{"type":"v1","addr":"172.21.15.82:6839","nonce":2084218047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.70:6837/3323713018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3323713018},{"type":"v1","addr":"172.21.15.70:6837","nonce":3323713018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6839/902276547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":902276547},{"type":"v1","addr":"172.21.15.70:6839","nonce":902276547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24446,24464,24442]},"id":1}]} 2024-05-03T17:23:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.782+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1728985200 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb5d40b2910 msgr2=0x7fb5d40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.782+0000 7fb5ed1fe700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb5d40b2910 0x7fb5d40b4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb5e80799b0 tx=0x7fb5d0008040 comp rx=0 tx=0).stop 2024-05-03T17:23:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.782+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1728985200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e806bbd0 msgr2=0x7fb5e81024a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.782+0000 7fb5ed1fe700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e806bbd0 0x7fb5e81024a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8003ac0 tx=0x7fb5d8003af0 comp rx=0 tx=0).stop 2024-05-03T17:23:36.783 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.782+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1728985200 shutdown_connections 2024-05-03T17:23:36.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.782+0000 7fb5ed1fe700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fb5d40b2910 0x7fb5d40b4dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:36.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.782+0000 7fb5ed1fe700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fb5e8109530 0x7fb5e8102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:36.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.782+0000 7fb5ed1fe700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fb5e806bbd0 0x7fb5e81024a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:36.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.782+0000 7fb5ed1fe700 1 --2- 172.21.15.67:0/1728985200 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fb5e806c590 0x7fb5e81029e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:36.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.782+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1728985200 >> 172.21.15.67:0/1728985200 conn(0x7fb5e8079340 msgr2=0x7fb5e8101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:36.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.782+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1728985200 shutdown_connections 2024-05-03T17:23:36.784 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:36.782+0000 7fb5ed1fe700 1 -- 172.21.15.67:0/1728985200 wait complete. 2024-05-03T17:23:36.784 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 25 2024-05-03T17:23:37.570 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T17:23:37.820 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:37.943 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:37 smithi067 ceph-mon[29731]: pgmap v386: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:37.943 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:37 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1728985200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:23:38.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:37 smithi082 ceph-mon[33535]: pgmap v386: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:38.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:37 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1728985200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:23:38.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:37 smithi070 ceph-mon[33250]: pgmap v386: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:38.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:37 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1728985200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:23:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.676+0000 7f66c4d4a700 1 -- 172.21.15.67:0/1086852236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c0109590 msgr2=0x7f66c0109970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.676+0000 7f66c4d4a700 1 --2- 172.21.15.67:0/1086852236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c0109590 0x7f66c0109970 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f66b00099c0 tx=0x7f66b0009cd0 comp rx=0 tx=0).stop 2024-05-03T17:23:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.676+0000 7f66c4d4a700 1 -- 172.21.15.67:0/1086852236 shutdown_connections 2024-05-03T17:23:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.676+0000 7f66c4d4a700 1 --2- 172.21.15.67:0/1086852236 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66c006c630 0x7f66c0110360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.676+0000 7f66c4d4a700 1 --2- 172.21.15.67:0/1086852236 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c0109590 0x7f66c0109970 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.676+0000 7f66c4d4a700 1 --2- 172.21.15.67:0/1086852236 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66c006bc70 0x7f66c006c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.676+0000 7f66c4d4a700 1 -- 172.21.15.67:0/1086852236 >> 172.21.15.67:0/1086852236 conn(0x7f66c0079440 msgr2=0x7f66c0079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.676+0000 7f66c4d4a700 1 -- 172.21.15.67:0/1086852236 shutdown_connections 2024-05-03T17:23:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.676+0000 7f66c4d4a700 1 -- 172.21.15.67:0/1086852236 wait complete. 2024-05-03T17:23:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.676+0000 7f66c4d4a700 1 Processor -- start 2024-05-03T17:23:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66c4d4a700 1 -- start start 2024-05-03T17:23:39.708 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66c4d4a700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66c006bc70 0x7f66c019f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66c4d4a700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66c006c630 0x7f66c019fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66c4d4a700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c0109590 0x7f66c0199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66c4d4a700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f66c0113690 con 0x7f66c0109590 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66c4d4a700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f66c0113510 con 0x7f66c006c630 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66c4d4a700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f66c0113810 con 0x7f66c006bc70 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66bed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c0109590 0x7f66c0199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66bed9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c0109590 0x7f66c0199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48504/0 (socket says 172.21.15.67:48504) 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66bed9d700 1 -- 172.21.15.67:0/2969320878 learned_addr learned my addr 172.21.15.67:0/2969320878 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66bdd9b700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66c006c630 0x7f66c019fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66be59c700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66c006bc70 0x7f66c019f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66bed9d700 1 -- 172.21.15.67:0/2969320878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66c006bc70 msgr2=0x7f66c019f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66bed9d700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66c006bc70 0x7f66c019f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66bed9d700 1 -- 172.21.15.67:0/2969320878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66c006c630 msgr2=0x7f66c019fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66bed9d700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66c006c630 0x7f66c019fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66bed9d700 1 -- 172.21.15.67:0/2969320878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c019a170 con 0x7f66c0109590 2024-05-03T17:23:39.709 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.677+0000 7f66bed9d700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c0109590 0x7f66c0199a10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f66b800ada0 tx=0x7f66b800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.678+0000 7f66af7fe700 1 -- 172.21.15.67:0/2969320878 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8017020 con 0x7f66c0109590 2024-05-03T17:23:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.678+0000 7f66af7fe700 1 -- 172.21.15.67:0/2969320878 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66b800cd80 con 0x7f66c0109590 2024-05-03T17:23:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.678+0000 7f66c4d4a700 1 -- 172.21.15.67:0/2969320878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c019a460 con 0x7f66c0109590 2024-05-03T17:23:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.678+0000 7f66af7fe700 1 -- 172.21.15.67:0/2969320878 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66b8005260 con 0x7f66c0109590 2024-05-03T17:23:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.678+0000 7f66c4d4a700 1 -- 172.21.15.67:0/2969320878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c01a66d0 con 0x7f66c0109590 2024-05-03T17:23:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.679+0000 7f66c4d4a700 1 -- 172.21.15.67:0/2969320878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c010ba20 con 0x7f66c0109590 2024-05-03T17:23:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.680+0000 7f66af7fe700 1 -- 172.21.15.67:0/2969320878 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f66b800a090 con 0x7f66c0109590 2024-05-03T17:23:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.681+0000 7f66af7fe700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f66a80b2910 0x7f66a80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.681+0000 7f66be59c700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f66a80b2910 0x7f66a80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.681+0000 7f66af7fe700 1 -- 172.21.15.67:0/2969320878 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f66b80e2030 con 0x7f66c0109590 2024-05-03T17:23:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.681+0000 7f66be59c700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f66a80b2910 0x7f66a80b4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f66c0100a60 tx=0x7f66b0005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:39.710 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.684+0000 7f66af7fe700 1 -- 172.21.15.67:0/2969320878 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f66b8021020 con 0x7f66c0109590 2024-05-03T17:23:39.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.899+0000 7f66c4d4a700 1 -- 172.21.15.67:0/2969320878 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f66c010b400 con 0x7f66c0109590 2024-05-03T17:23:39.900 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:39 smithi067 ceph-mon[29731]: pgmap v387: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:39.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.901+0000 7f66af7fe700 1 -- 172.21.15.67:0/2969320878 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22635 (secure 0 0 0) 0x7f66b80a9c70 con 0x7f66c0109590 2024-05-03T17:23:39.902 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:23:39.902 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":81,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","created":"2024-05-03T17:12:11.709381+0000","modified":"2024-05-03T17:21:07.584731+0000","last_up_change":"2024-05-03T17:19:29.310797+0000","last_in_change":"2024-05-03T17:19:08.848654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T17:15:36.413266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T17:20:31.546843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T17:20:35.261082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80962eb1-a506-49bb-a528-dee035eb4322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6803","nonce":1754365741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6805","nonce":1754365741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6809","nonce":1754365741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6807","nonce":1754365741}]},"public_addr":"172.21.15.67:6803/1754365741","cluster_addr":"172.21.15.67:6805/1754365741","heartbeat_back_addr":"172.21.15.67:6809/1754365741","heartbeat_front_addr":"172.21.15.67:6807/1754365741","state":["exists","up"]},{"osd":1,"uuid":"d7721d5f-4373-4cb2-bb57-bc368d6bb818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6801","nonce":1006339176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6803","nonce":1006339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6807","nonce":1006339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6805","nonce":1006339176}]},"public_addr":"172.21.15.70:6801/1006339176","cluster_addr":"172.21.15.70:6803/1006339176","heartbeat_back_addr":"172.21.15.70:6807/1006339176","heartbeat_front_addr":"172.21.15.70:6805/1006339176","state":["exists","up"]},{"osd":2,"uuid":"9992fd98-821b-40ce-b051-89befbfd87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6801","nonce":1888803264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6803","nonce":1888803264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6807","nonce":1888803264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6805","nonce":1888803264}]},"public_addr":"172.21.15.82:6801/1888803264","cluster_addr":"172.21.15.82:6803/1888803264","heartbeat_back_addr":"172.21.15.82:6807/1888803264","heartbeat_front_addr":"172.21.15.82:6805/1888803264","state":["exists","up"]},{"osd":3,"uuid":"14bfe6a6-0bc8-4766-8d36-fbaecc61a396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6811","nonce":313717723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6813","nonce":313717723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6817","nonce":313717723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6815","nonce":313717723}]},"public_addr":"172.21.15.67:6811/313717723","cluster_addr":"172.21.15.67:6813/313717723","heartbeat_back_addr":"172.21.15.67:6817/313717723","heartbeat_front_addr":"172.21.15.67:6815/313717723","state":["exists","up"]},{"osd":4,"uuid":"5a44aeb8-aabf-4824-be65-2c7170dbee52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6809","nonce":3526300828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6811","nonce":3526300828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6815","nonce":3526300828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6813","nonce":3526300828}]},"public_addr":"172.21.15.70:6809/3526300828","cluster_addr":"172.21.15.70:6811/3526300828","heartbeat_back_addr":"172.21.15.70:6815/3526300828","heartbeat_front_addr":"172.21.15.70:6813/3526300828","state":["exists","up"]},{"osd":5,"uuid":"a6ac56df-466e-4096-8fbf-835f5b544297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6809","nonce":3754223290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6811","nonce":3754223290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6815","nonce":3754223290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6813","nonce":3754223290}]},"public_addr":"172.21.15.82:6809/3754223290","cluster_addr":"172.21.15.82:6811/3754223290","heartbeat_back_addr":"172.21.15.82:6815/3754223290","heartbeat_front_addr":"172.21.15.82:6813/3754223290","state":["exists","up"]},{"osd":6,"uuid":"4d8a5e8a-66ac-462e-b960-24c0ee248fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6819","nonce":4033769161}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6821","nonce":4033769161}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6825","nonce":4033769161}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6823","nonce":4033769161}]},"public_addr":"172.21.15.67:6819/4033769161","cluster_addr":"172.21.15.67:6821/4033769161","heartbeat_back_addr":"172.21.15.67:6825/4033769161","heartbeat_front_addr":"172.21.15.67:6823/4033769161","state":["exists","up"]},{"osd":7,"uuid":"eb983837-8172-4802-9843-6360cda8f4be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6817","nonce":3366146173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6819","nonce":3366146173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6823","nonce":3366146173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6821","nonce":3366146173}]},"public_addr":"172.21.15.70:6817/3366146173","cluster_addr":"172.21.15.70:6819/3366146173","heartbeat_back_addr":"172.21.15.70:6823/3366146173","heartbeat_front_addr":"172.21.15.70:6821/3366146173","state":["exists","up"]},{"osd":8,"uuid":"c9d4ce56-e940-48d1-a9b5-437139e3fed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6817","nonce":347699286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6819","nonce":347699286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6823","nonce":347699286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6821","nonce":347699286}]},"public_addr":"172.21.15.82:6817/347699286","cluster_addr":"172.21.15.82:6819/347699286","heartbeat_back_addr":"172.21.15.82:6823/347699286","heartbeat_front_addr":"172.21.15.82:6821/347699286","state":["exists","up"]},{"osd":9,"uuid":"09596d19-f2d7-485e-a028-9ce5468c6651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6827","nonce":2600095198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6829","nonce":2600095198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6833","nonce":2600095198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6831","nonce":2600095198}]},"public_addr":"172.21.15.67:6827/2600095198","cluster_addr":"172.21.15.67:6829/2600095198","heartbeat_back_addr":"172.21.15.67:6833/2600095198","heartbeat_front_addr":"172.21.15.67:6831/2600095198","state":["exists","up"]},{"osd":10,"uuid":"ec4cffbc-0127-47e5-8176-0972f74a5514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6825","nonce":2897583629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6827","nonce":2897583629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6831","nonce":2897583629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6829","nonce":2897583629}]},"public_addr":"172.21.15.70:6825/2897583629","cluster_addr":"172.21.15.70:6827/2897583629","heartbeat_back_addr":"172.21.15.70:6831/2897583629","heartbeat_front_addr":"172.21.15.70:6829/2897583629","state":["exists","up"]},{"osd":11,"uuid":"a8acfb0b-98e8-49e1-a114-f33aa8160b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6825","nonce":4082656149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6827","nonce":4082656149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6831","nonce":4082656149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6829","nonce":4082656149}]},"public_addr":"172.21.15.82:6825/4082656149","cluster_addr":"172.21.15.82:6827/4082656149","heartbeat_back_addr":"172.21.15.82:6831/4082656149","heartbeat_front_addr":"172.21.15.82:6829/4082656149","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:14:45.596665+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:09.538408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:33.608945+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:59.803102+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:25.187966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:49.892160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:16.575973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:42.115955+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:07.655558+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:35.356779+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:01.144695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:27.577513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/2131289004":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/3335149382":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/992151181":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2614686067":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/3137329076":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2323686292":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/2579128777":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/3335149382":"2024-05-04T17:13:06.324002+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T17:23:39.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.903+0000 7f66c4d4a700 1 -- 172.21.15.67:0/2969320878 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f66a80b2910 msgr2=0x7f66a80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.903+0000 7f66c4d4a700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f66a80b2910 0x7f66a80b4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f66c0100a60 tx=0x7f66b0005c40 comp rx=0 tx=0).stop 2024-05-03T17:23:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.903+0000 7f66c4d4a700 1 -- 172.21.15.67:0/2969320878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c0109590 msgr2=0x7f66c0199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.903+0000 7f66c4d4a700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c0109590 0x7f66c0199a10 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f66b800ada0 tx=0x7f66b800c330 comp rx=0 tx=0).stop 2024-05-03T17:23:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.903+0000 7f66c4d4a700 1 -- 172.21.15.67:0/2969320878 shutdown_connections 2024-05-03T17:23:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.903+0000 7f66c4d4a700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f66a80b2910 0x7f66a80b4dd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.903+0000 7f66c4d4a700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f66c0109590 0x7f66c0199a10 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.903+0000 7f66c4d4a700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f66c006bc70 0x7f66c019f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:39.904 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.903+0000 7f66c4d4a700 1 --2- 172.21.15.67:0/2969320878 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f66c006c630 0x7f66c019fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.903+0000 7f66c4d4a700 1 -- 172.21.15.67:0/2969320878 >> 172.21.15.67:0/2969320878 conn(0x7f66c0079440 msgr2=0x7f66c0100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.903+0000 7f66c4d4a700 1 -- 172.21.15.67:0/2969320878 shutdown_connections 2024-05-03T17:23:39.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:39.903+0000 7f66c4d4a700 1 -- 172.21.15.67:0/2969320878 wait complete. 2024-05-03T17:23:40.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:39 smithi082 ceph-mon[33535]: pgmap v387: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:40.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:39 smithi070 ceph-mon[33250]: pgmap v387: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:40.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-03T17:20:39.322107+0000', 'modified': '2024-05-03T17:21:01.753515+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24464, 'mds_1': 24446, 'mds_2': 24442}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24421': {'gid': 24421, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6835/1568454842', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6834', 'nonce': 1568454842}, {'type': 'v1', 'addr': '172.21.15.82:6835', 'nonce': 1568454842}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24425': {'gid': 24425, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.70:6835/1966549782', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6834', 'nonce': 1966549782}, {'type': 'v1', 'addr': '172.21.15.70:6835', 'nonce': 1966549782}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24427': {'gid': 24427, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.82:6837/370282857', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6836', 'nonce': 370282857}, {'type': 'v1', 'addr': '172.21.15.82:6837', 'nonce': 370282857}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.82:6839/2084218047', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.82:6838', 'nonce': 2084218047}, {'type': 'v1', 'addr': '172.21.15.82:6839', 'nonce': 2084218047}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.70:6837/3323713018', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6836', 'nonce': 3323713018}, {'type': 'v1', 'addr': '172.21.15.70:6837', 'nonce': 3323713018}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24464': {'gid': 24464, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.70:6839/902276547', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.70:6838', 'nonce': 902276547}, {'type': 'v1', 'addr': '172.21.15.70:6839', 'nonce': 902276547}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24464, 'qdb_cluster': [24446, 24464, 24442]} 2024-05-03T17:23:40.572 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-03T17:23:40.572 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell mds.i status 2024-05-03T17:23:40.831 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:40.862 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:40 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2969320878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:23:41.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:40 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2969320878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:23:41.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:40 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2969320878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:23:42.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:41 smithi082 ceph-mon[33535]: pgmap v388: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:42.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:41 smithi070 ceph-mon[33250]: pgmap v388: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:42.060 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:41 smithi067 ceph-mon[29731]: pgmap v388: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.636+0000 7f7c58256700 1 -- 172.21.15.67:0/3305447243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50102820 msgr2=0x7f7c50110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.636+0000 7f7c58256700 1 --2- 172.21.15.67:0/3305447243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50102820 0x7f7c50110290 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800b0a0 tx=0x7f7c4800b3b0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.636+0000 7f7c58256700 1 -- 172.21.15.67:0/3305447243 shutdown_connections 2024-05-03T17:23:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.636+0000 7f7c58256700 1 --2- 172.21.15.67:0/3305447243 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50102820 0x7f7c50110290 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.636+0000 7f7c58256700 1 --2- 172.21.15.67:0/3305447243 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c50108580 0x7f7c50108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.636+0000 7f7c58256700 1 --2- 172.21.15.67:0/3305447243 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c50101e60 0x7f7c501022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.636+0000 7f7c58256700 1 -- 172.21.15.67:0/3305447243 >> 172.21.15.67:0/3305447243 conn(0x7f7c500776c0 msgr2=0x7f7c50077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.636+0000 7f7c58256700 1 -- 172.21.15.67:0/3305447243 shutdown_connections 2024-05-03T17:23:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.636+0000 7f7c58256700 1 -- 172.21.15.67:0/3305447243 wait complete. 2024-05-03T17:23:42.639 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.637+0000 7f7c58256700 1 Processor -- start 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.637+0000 7f7c58256700 1 -- start start 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.637+0000 7f7c58256700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c50101e60 0x7f7c501a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.637+0000 7f7c58256700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c50102820 0x7f7c501a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.637+0000 7f7c58256700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50108580 0x7f7c5019bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.637+0000 7f7c58256700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f7c50113690 con 0x7f7c50108580 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.637+0000 7f7c58256700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f7c50113510 con 0x7f7c50101e60 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.637+0000 7f7c58256700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f7c50113810 con 0x7f7c50102820 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.637+0000 7f7c567f3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50108580 0x7f7c5019bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.637+0000 7f7c567f3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50108580 0x7f7c5019bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48524/0 (socket says 172.21.15.67:48524) 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.637+0000 7f7c567f3700 1 -- 172.21.15.67:0/1624489849 learned_addr learned my addr 172.21.15.67:0/1624489849 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c50101e60 0x7f7c501a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c557f1700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c50102820 0x7f7c501a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c567f3700 1 -- 172.21.15.67:0/1624489849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c50102820 msgr2=0x7f7c501a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c567f3700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c50102820 0x7f7c501a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.640 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c567f3700 1 -- 172.21.15.67:0/1624489849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c50101e60 msgr2=0x7f7c501a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c567f3700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c50101e60 0x7f7c501a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c567f3700 1 -- 172.21.15.67:0/1624489849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c5019c230 con 0x7f7c50108580 2024-05-03T17:23:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c557f1700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c50102820 0x7f7c501a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:23:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c50101e60 0x7f7c501a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:23:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c567f3700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50108580 0x7f7c5019bad0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800b830 tx=0x7f7c48004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c46ffd700 1 -- 172.21.15.67:0/1624489849 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48021070 con 0x7f7c50108580 2024-05-03T17:23:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c46ffd700 1 -- 172.21.15.67:0/1624489849 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c48003c00 con 0x7f7c50108580 2024-05-03T17:23:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c58256700 1 -- 172.21.15.67:0/1624489849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c5019c4c0 con 0x7f7c50108580 2024-05-03T17:23:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c46ffd700 1 -- 172.21.15.67:0/1624489849 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48004730 con 0x7f7c50108580 2024-05-03T17:23:42.641 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.638+0000 7f7c58256700 1 -- 172.21.15.67:0/1624489849 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c501a87e0 con 0x7f7c50108580 2024-05-03T17:23:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.641+0000 7f7c46ffd700 1 -- 172.21.15.67:0/1624489849 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f7c48007a00 con 0x7f7c50108580 2024-05-03T17:23:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.641+0000 7f7c46ffd700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f7c3c0b2910 0x7f7c3c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.642+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f7c3c0b2910 0x7f7c3c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.642 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.642+0000 7f7c46ffd700 1 -- 172.21.15.67:0/1624489849 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f7c4800f020 con 0x7f7c50108580 2024-05-03T17:23:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.642+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f7c3c0b2910 0x7f7c3c0b4dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7c50078c40 tx=0x7f7c4c00b5f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.642+0000 7f7c62ae9b80 1 Processor -- start 2024-05-03T17:23:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.642+0000 7f7c62ae9b80 1 -- start start 2024-05-03T17:23:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c62ae9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b6740 0x55b0ba3b6b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c62ae9b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 0x55b0ba3b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c62ae9b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 0x55b0ba3b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c62ae9b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b0ba3048f0 con 0x55b0ba3b7060 2024-05-03T17:23:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c62ae9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b0ba269d50 con 0x55b0ba3b6740 2024-05-03T17:23:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c62ae9b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b0ba3008b0 con 0x55b0ba3b2480 2024-05-03T17:23:42.644 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c557f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 0x55b0ba3b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c557f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 0x55b0ba3b1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48534/0 (socket says 172.21.15.67:48534) 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c557f1700 1 -- 172.21.15.67:0/555710620 learned_addr learned my addr 172.21.15.67:0/555710620 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c567f3700 1 --2- 172.21.15.67:0/555710620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 0x55b0ba3b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/555710620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b6740 0x55b0ba3b6b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c557f1700 1 -- 172.21.15.67:0/555710620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 msgr2=0x55b0ba3b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c557f1700 1 --2- 172.21.15.67:0/555710620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 0x55b0ba3b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c557f1700 1 -- 172.21.15.67:0/555710620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b6740 msgr2=0x55b0ba3b6b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c557f1700 1 --2- 172.21.15.67:0/555710620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b6740 0x55b0ba3b6b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c557f1700 1 -- 172.21.15.67:0/555710620 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0ba251880 con 0x55b0ba3b7060 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/555710620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b6740 0x55b0ba3b6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c557f1700 1 --2- 172.21.15.67:0/555710620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 0x55b0ba3b1f40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000bda0 tx=0x7f7c4000f640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f898f0042055cc server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.643+0000 7f7c3bfff700 1 -- 172.21.15.67:0/555710620 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40015070 con 0x55b0ba3b7060 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.644+0000 7f7c3bfff700 1 -- 172.21.15.67:0/555710620 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c4000fda0 con 0x55b0ba3b7060 2024-05-03T17:23:42.645 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.644+0000 7f7c3bfff700 1 -- 172.21.15.67:0/555710620 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40011680 con 0x55b0ba3b7060 2024-05-03T17:23:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.644+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/555710620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 msgr2=0x55b0ba3b1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.644+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/555710620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 0x55b0ba3b1f40 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000bda0 tx=0x7f7c4000f640 comp rx=0 tx=0).stop 2024-05-03T17:23:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.644+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/555710620 shutdown_connections 2024-05-03T17:23:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.644+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/555710620 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 0x55b0ba3b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.649 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.644+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/555710620 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b6740 0x55b0ba3b6b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.644+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/555710620 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 0x55b0ba3b1f40 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.644+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/555710620 >> 172.21.15.67:0/555710620 conn(0x55b0ba2deb20 msgr2=0x55b0ba3ac9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.644+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/555710620 shutdown_connections 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.644+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/555710620 wait complete. 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.645+0000 7f7c62ae9b80 1 -- start start 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.645+0000 7f7c62ae9b80 1 Processor -- start 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.645+0000 7f7c62ae9b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 0x55b0ba3bdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.645+0000 7f7c62ae9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b6740 0x55b0ba3bb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.645+0000 7f7c62ae9b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 0x55b0ba3bb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.645+0000 7f7c62ae9b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b0ba3beba0 con 0x55b0ba3b7060 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.645+0000 7f7c62ae9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b0ba2bde80 con 0x55b0ba3b6740 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.645+0000 7f7c62ae9b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b0ba3bed20 con 0x55b0ba3b2480 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.645+0000 7f7c62ae9b80 10 client.0 _ll_register_callbacks cb 0x55b0ba298560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.645+0000 7f7c567f3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 0x55b0ba3bb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.650 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.645+0000 7f7c567f3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 0x55b0ba3bb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48550/0 (socket says 172.21.15.67:48550) 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.645+0000 7f7c567f3700 1 -- 172.21.15.67:0/2827583901 learned_addr learned my addr 172.21.15.67:0/2827583901 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c557f1700 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b6740 0x55b0ba3bb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 0x55b0ba3bdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c567f3700 1 -- 172.21.15.67:0/2827583901 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 msgr2=0x55b0ba3bdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c567f3700 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 0x55b0ba3bdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c567f3700 1 -- 172.21.15.67:0/2827583901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b6740 msgr2=0x55b0ba3bb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c567f3700 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b6740 0x55b0ba3bb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c567f3700 1 -- 172.21.15.67:0/2827583901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0ba2bb710 con 0x55b0ba3b7060 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c567f3700 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 0x55b0ba3bb6b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7c480e1490 tx=0x7f7c480aab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e1a6fe39ae718f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c397fa700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c397fa700 1 -- 172.21.15.67:0/2827583901 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c480af3b0 con 0x55b0ba3b7060 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c397fa700 1 -- 172.21.15.67:0/2827583901 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f7c480e16b0 con 0x55b0ba3b7060 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2827583901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c4000fda0 con 0x55b0ba3b7060 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c397fa700 1 -- 172.21.15.67:0/2827583901 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7c480b3dd0 con 0x55b0ba3b7060 2024-05-03T17:23:42.651 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c62ae9b80 10 client.15045 fetch_fsmap learned FSMap version 25 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2827583901 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b0ba3c93d0 con 0x55b0ba3b7060 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.646+0000 7f7c397fa700 1 -- 172.21.15.67:0/2827583901 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c4000fda0 con 0x55b0ba3b7060 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.647+0000 7f7c397fa700 1 -- 172.21.15.67:0/2827583901 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f7c48011070 con 0x55b0ba3b7060 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.647+0000 7f7c62ae9b80 10 client.15045 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.647+0000 7f7c62ae9b80 10 client.15045 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.647+0000 7f7c62ae9b80 20 client.15045 populate_metadata read hostname 'smithi067' 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.647+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55b0ba3cb990 0x55b0ba3cce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.647+0000 7f7c62ae9b80 4 client.15045 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.647+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2827583901 --> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0ba3122d0 con 0x55b0ba3cb990 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.647+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55b0ba3cb990 0x55b0ba3cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.647+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55b0ba3cb990 0x55b0ba3cce00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5e3fc4775ff95851 server_cookie=e503136545d4073d in_seq=0 out_seq=0 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.647+0000 7f7c397fa700 10 client.15045 ms_handle_connect on v2:172.21.15.82:6838/2084218047 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.648+0000 7f7c397fa700 1 -- 172.21.15.67:0/2827583901 <== mds.2 v2:172.21.15.82:6838/2084218047 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b0ba3122d0 con 0x55b0ba3cb990 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.648+0000 7f7c397fa700 10 client.15045 handle_command_reply: tid=0 2024-05-03T17:23:42.652 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.648+0000 7f7c62ae9b80 1 client.15045 shutdown 2024-05-03T17:23:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.648+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2827583901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 msgr2=0x55b0ba3bb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.648+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 0x55b0ba3bb6b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f7c480e1490 tx=0x7f7c480aab90 comp rx=0 tx=0).stop 2024-05-03T17:23:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.648+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2827583901 shutdown_connections 2024-05-03T17:23:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.648+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55b0ba3cb990 0x55b0ba3cce00 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.648+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b7060 0x55b0ba3bb6b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.648+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 0x55b0ba3bdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.648+0000 7f7c397fa700 0 client.15045 ms_handle_reset on v2:172.21.15.82:6838/2084218047 2024-05-03T17:23:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.648+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2827583901 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b6740 0x55b0ba3bb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.648+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2827583901 >> 172.21.15.67:0/2827583901 conn(0x55b0ba2deb20 msgr2=0x55b0ba3b7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.649+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2827583901 shutdown_connections 2024-05-03T17:23:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.649+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2827583901 wait complete. 2024-05-03T17:23:42.653 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.649+0000 7f7c62ae9b80 20 client.15045 trim_cache size 0 max 16384 2024-05-03T17:23:42.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c62ae9b80 1 Processor -- start 2024-05-03T17:23:42.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c62ae9b80 1 -- start start 2024-05-03T17:23:42.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c62ae9b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 0x55b0ba3ac810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.671 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c62ae9b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b6740 0x55b0ba3aff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c62ae9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b7060 0x55b0ba3b0470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c62ae9b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b0ba3cb140 con 0x55b0ba3b6740 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c62ae9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b0ba3cb810 con 0x55b0ba3b7060 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c62ae9b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b0ba3cbee0 con 0x55b0ba3b2480 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c557f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b6740 0x55b0ba3aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c557f1700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b6740 0x55b0ba3aff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48556/0 (socket says 172.21.15.67:48556) 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c557f1700 1 -- 172.21.15.67:0/2083315607 learned_addr learned my addr 172.21.15.67:0/2083315607 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c567f3700 1 --2- 172.21.15.67:0/2083315607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b7060 0x55b0ba3b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/2083315607 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 0x55b0ba3ac810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c557f1700 1 -- 172.21.15.67:0/2083315607 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 msgr2=0x55b0ba3ac810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c557f1700 1 --2- 172.21.15.67:0/2083315607 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 0x55b0ba3ac810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c557f1700 1 -- 172.21.15.67:0/2083315607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b7060 msgr2=0x55b0ba3b0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c557f1700 1 --2- 172.21.15.67:0/2083315607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b7060 0x55b0ba3b0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c557f1700 1 -- 172.21.15.67:0/2083315607 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0ba3b96e0 con 0x55b0ba3b6740 2024-05-03T17:23:42.672 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.670+0000 7f7c567f3700 1 --2- 172.21.15.67:0/2083315607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b7060 0x55b0ba3b0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:23:42.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.671+0000 7f7c557f1700 1 --2- 172.21.15.67:0/2083315607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b6740 0x55b0ba3aff30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000f990 tx=0x7f7c4000fde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28d05ec26cb998e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:42.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.671+0000 7f7c3b7fe700 1 -- 172.21.15.67:0/2083315607 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40029070 con 0x55b0ba3b6740 2024-05-03T17:23:42.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.671+0000 7f7c3b7fe700 1 -- 172.21.15.67:0/2083315607 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c40004710 con 0x55b0ba3b6740 2024-05-03T17:23:42.673 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.671+0000 7f7c3b7fe700 1 -- 172.21.15.67:0/2083315607 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40015040 con 0x55b0ba3b6740 2024-05-03T17:23:42.676 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.671+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2083315607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b6740 msgr2=0x55b0ba3aff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.671+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2083315607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b6740 0x55b0ba3aff30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000f990 tx=0x7f7c4000fde0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.671+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2083315607 shutdown_connections 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.671+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2083315607 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b7060 0x55b0ba3b0470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.671+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2083315607 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b2480 0x55b0ba3ac810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.671+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2083315607 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b6740 0x55b0ba3aff30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.671+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2083315607 >> 172.21.15.67:0/2083315607 conn(0x55b0ba2deb20 msgr2=0x55b0ba3b9a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.672+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2083315607 shutdown_connections 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.672+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2083315607 wait complete. 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.672+0000 7f7c62ae9b80 1 -- start start 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.672+0000 7f7c62ae9b80 1 Processor -- start 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c62ae9b80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b2480 0x55b0ba3d5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c62ae9b80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b6740 0x55b0ba3d57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c62ae9b80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b7060 0x55b0ba3d5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.677 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c62ae9b80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b0ba3ccc80 con 0x55b0ba3b2480 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c62ae9b80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b0ba3cc5b0 con 0x55b0ba3b7060 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c62ae9b80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b0ba3cd350 con 0x55b0ba3b6740 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c62ae9b80 10 client.0 _ll_register_callbacks cb 0x55b0ba298560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c55ff2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b2480 0x55b0ba3d5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c55ff2700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b2480 0x55b0ba3d5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48560/0 (socket says 172.21.15.67:48560) 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c55ff2700 1 -- 172.21.15.67:0/2516719777 learned_addr learned my addr 172.21.15.67:0/2516719777 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c557f1700 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b6740 0x55b0ba3d57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c567f3700 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b7060 0x55b0ba3d5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c55ff2700 1 -- 172.21.15.67:0/2516719777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b6740 msgr2=0x55b0ba3d57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b6740 0x55b0ba3d57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c55ff2700 1 -- 172.21.15.67:0/2516719777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b7060 msgr2=0x55b0ba3d5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b7060 0x55b0ba3d5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c55ff2700 1 -- 172.21.15.67:0/2516719777 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0ba3d6730 con 0x55b0ba3b2480 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c55ff2700 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b2480 0x55b0ba3d5270 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00b9f0 tx=0x7f7c4c0047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=289cf4fa8ebbacde server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:42.678 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c397fa700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c397fa700 1 -- 172.21.15.67:0/2516719777 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4c02d070 con 0x55b0ba3b2480 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2516719777 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c40004710 con 0x55b0ba3b2480 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.673+0000 7f7c397fa700 1 -- 172.21.15.67:0/2516719777 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f7c4c029530 con 0x55b0ba3b2480 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.674+0000 7f7c397fa700 1 -- 172.21.15.67:0/2516719777 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7c4c039050 con 0x55b0ba3b2480 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.674+0000 7f7c62ae9b80 10 client.15051 fetch_fsmap learned FSMap version 25 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.674+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2516719777 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b0ba3b7ed0 con 0x55b0ba3b2480 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.674+0000 7f7c397fa700 1 -- 172.21.15.67:0/2516719777 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c40004710 con 0x55b0ba3b2480 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.674+0000 7f7c397fa700 1 -- 172.21.15.67:0/2516719777 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f7c4c011ac0 con 0x55b0ba3b2480 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.674+0000 7f7c62ae9b80 10 client.15051 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.674+0000 7f7c62ae9b80 10 client.15051 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.674+0000 7f7c62ae9b80 20 client.15051 populate_metadata read hostname 'smithi067' 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.674+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55b0ba43c4c0 0x55b0ba43b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.674+0000 7f7c62ae9b80 4 client.15051 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.674+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2516719777 --> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] -- command(tid 0: {"prefix": "status"}) -- 0x55b0ba30c370 con 0x55b0ba43c4c0 2024-05-03T17:23:42.679 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.675+0000 7f7c557f1700 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55b0ba43c4c0 0x55b0ba43b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.675+0000 7f7c557f1700 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55b0ba43c4c0 0x55b0ba43b1d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=770147fa8e35c039 server_cookie=2ed3717cbc3236d1 in_seq=0 out_seq=0 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.675+0000 7f7c397fa700 10 client.15051 ms_handle_connect on v2:172.21.15.82:6838/2084218047 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.675+0000 7f7c397fa700 1 -- 172.21.15.67:0/2516719777 <== mds.2 v2:172.21.15.82:6838/2084218047 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55b0ba30c370 con 0x55b0ba43c4c0 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.675+0000 7f7c397fa700 10 client.15051 handle_command_reply: tid=0 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.675+0000 7f7c62ae9b80 1 client.15051 shutdown 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.675+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2516719777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b2480 msgr2=0x55b0ba3d5270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.675+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b2480 0x55b0ba3d5270 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7c4c00b9f0 tx=0x7f7c4c0047c0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2516719777 shutdown_connections 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.82:6838/2084218047,v1:172.21.15.82:6839/2084218047] conn(0x55b0ba43c4c0 0x55b0ba43b1d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b0ba3b7060 0x55b0ba3d5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b0ba3b2480 0x55b0ba3d5270 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c397fa700 0 client.15051 ms_handle_reset on v2:172.21.15.82:6838/2084218047 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c62ae9b80 1 --2- 172.21.15.67:0/2516719777 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b0ba3b6740 0x55b0ba3d57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2516719777 >> 172.21.15.67:0/2516719777 conn(0x55b0ba2deb20 msgr2=0x55b0ba3bc7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2516719777 shutdown_connections 2024-05-03T17:23:42.680 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c62ae9b80 1 -- 172.21.15.67:0/2516719777 wait complete. 2024-05-03T17:23:42.681 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c62ae9b80 20 client.15051 trim_cache size 0 max 16384 2024-05-03T17:23:42.681 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T17:23:42.681 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "20c8e22e-0970-11ef-bc97-c7b262605968", 2024-05-03T17:23:42.681 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 2, 2024-05-03T17:23:42.681 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24442, 2024-05-03T17:23:42.681 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-03T17:23:42.681 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-03T17:23:42.681 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-03T17:23:42.681 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 167.125007823, 2024-05-03T17:23:42.681 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-03T17:23:42.681 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 81, 2024-05-03T17:23:42.682 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 79, 2024-05-03T17:23:42.682 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 210.076055712 2024-05-03T17:23:42.682 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T17:23:42.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c58256700 1 -- 172.21.15.67:0/1624489849 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f7c3c0b2910 msgr2=0x7f7c3c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:42.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c58256700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f7c3c0b2910 0x7f7c3c0b4dd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7c50078c40 tx=0x7f7c4c00b5f0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.682 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c58256700 1 -- 172.21.15.67:0/1624489849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50108580 msgr2=0x7f7c5019bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c58256700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50108580 0x7f7c5019bad0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800b830 tx=0x7f7c48004d90 comp rx=0 tx=0).stop 2024-05-03T17:23:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c58256700 1 -- 172.21.15.67:0/1624489849 shutdown_connections 2024-05-03T17:23:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c58256700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f7c3c0b2910 0x7f7c3c0b4dd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c58256700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7c50108580 0x7f7c5019bad0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c58256700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f7c50101e60 0x7f7c501a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c58256700 1 --2- 172.21.15.67:0/1624489849 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f7c50102820 0x7f7c501a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.676+0000 7f7c58256700 1 -- 172.21.15.67:0/1624489849 >> 172.21.15.67:0/1624489849 conn(0x7f7c500776c0 msgr2=0x7f7c50100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.677+0000 7f7c58256700 1 -- 172.21.15.67:0/1624489849 shutdown_connections 2024-05-03T17:23:42.683 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:42.677+0000 7f7c58256700 1 -- 172.21.15.67:0/1624489849 wait complete. 2024-05-03T17:23:43.384 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell mds.h status 2024-05-03T17:23:43.636 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:44.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:43 smithi082 ceph-mon[33535]: pgmap v389: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:44.040 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:43 smithi067 ceph-mon[29731]: pgmap v389: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:44.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:43 smithi070 ceph-mon[33250]: pgmap v389: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:44.879 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:23:44.879 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:44.879 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:44.879 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:44.880 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:44.880 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:44.880 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:44 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:23:45.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:44 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:23:45.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:44 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:45.420 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.420+0000 7f8a558aa700 1 -- 172.21.15.67:0/3722003034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a5006ba10 msgr2=0x7f8a5006be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.420+0000 7f8a558aa700 1 --2- 172.21.15.67:0/3722003034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a5006ba10 0x7f8a5006be90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f8a38009990 tx=0x7f8a38009ca0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.420+0000 7f8a558aa700 1 -- 172.21.15.67:0/3722003034 shutdown_connections 2024-05-03T17:23:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.420+0000 7f8a558aa700 1 --2- 172.21.15.67:0/3722003034 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8a5006c3d0 0x7f8a50110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.420+0000 7f8a558aa700 1 --2- 172.21.15.67:0/3722003034 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a50109500 0x7f8a501098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.420+0000 7f8a558aa700 1 --2- 172.21.15.67:0/3722003034 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a5006ba10 0x7f8a5006be90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.420+0000 7f8a558aa700 1 -- 172.21.15.67:0/3722003034 >> 172.21.15.67:0/3722003034 conn(0x7f8a50079310 msgr2=0x7f8a50079720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.420+0000 7f8a558aa700 1 -- 172.21.15.67:0/3722003034 shutdown_connections 2024-05-03T17:23:45.421 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.420+0000 7f8a558aa700 1 -- 172.21.15.67:0/3722003034 wait complete. 2024-05-03T17:23:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.421+0000 7f8a558aa700 1 Processor -- start 2024-05-03T17:23:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.421+0000 7f8a558aa700 1 -- start start 2024-05-03T17:23:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.421+0000 7f8a558aa700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8a5006ba10 0x7f8a5019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.421+0000 7f8a558aa700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a5006c3d0 0x7f8a5019fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.421+0000 7f8a558aa700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a50109500 0x7f8a501999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.423 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.421+0000 7f8a558aa700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f8a50113630 con 0x7f8a50109500 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.421+0000 7f8a558aa700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f8a501134b0 con 0x7f8a5006c3d0 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.421+0000 7f8a558aa700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f8a501137b0 con 0x7f8a5006ba10 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.421+0000 7f8a4f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a50109500 0x7f8a501999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.421+0000 7f8a4f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a50109500 0x7f8a501999f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48582/0 (socket says 172.21.15.67:48582) 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.421+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/1394140693 learned_addr learned my addr 172.21.15.67:0/1394140693 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.421+0000 7f8a4e7fc700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a5006c3d0 0x7f8a5019fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a4effd700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8a5006ba10 0x7f8a5019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/1394140693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8a5006ba10 msgr2=0x7f8a5019f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a4f7fe700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8a5006ba10 0x7f8a5019f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/1394140693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a5006c3d0 msgr2=0x7f8a5019fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a4f7fe700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a5006c3d0 0x7f8a5019fd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/1394140693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5019a150 con 0x7f8a50109500 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a4effd700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8a5006ba10 0x7f8a5019f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a4f7fe700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a50109500 0x7f8a501999f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000d980 tx=0x7f8a4000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a548a8700 1 -- 172.21.15.67:0/1394140693 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a40017070 con 0x7f8a50109500 2024-05-03T17:23:45.424 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a548a8700 1 -- 172.21.15.67:0/1394140693 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a40005020 con 0x7f8a50109500 2024-05-03T17:23:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a558aa700 1 -- 172.21.15.67:0/1394140693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5019a440 con 0x7f8a50109500 2024-05-03T17:23:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a548a8700 1 -- 172.21.15.67:0/1394140693 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a400099b0 con 0x7f8a50109500 2024-05-03T17:23:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.422+0000 7f8a558aa700 1 -- 172.21.15.67:0/1394140693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a501a6700 con 0x7f8a50109500 2024-05-03T17:23:45.425 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.425+0000 7f8a548a8700 1 -- 172.21.15.67:0/1394140693 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f8a40009b50 con 0x7f8a50109500 2024-05-03T17:23:45.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.425+0000 7f8a548a8700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8a3c0b2990 0x7f8a3c0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a4effd700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8a3c0b2990 0x7f8a3c0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.427 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a548a8700 1 -- 172.21.15.67:0/1394140693 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f8a400e21b0 con 0x7f8a50109500 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a6013db80 1 Processor -- start 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a6013db80 1 -- start start 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a6013db80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8f5c10 0x55b63a8f5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a6013db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8f9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a6013db80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 0x55b63a8fa690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a6013db80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b63a848810 con 0x55b63a8fa230 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a6013db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b63a7e7770 con 0x55b63a8f6530 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a6013db80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b63a8447d0 con 0x55b63a8f5c10 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a4effd700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8a3c0b2990 0x7f8a3c0b4e50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8a5019ab30 tx=0x7f8a4800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a4f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 0x55b63a8fa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a4f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 0x55b63a8fa690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48588/0 (socket says 172.21.15.67:48588) 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/481106964 learned_addr learned my addr 172.21.15.67:0/481106964 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a4effd700 1 --2- 172.21.15.67:0/481106964 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8f5c10 0x55b63a8f5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.428 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.426+0000 7f8a4e7fc700 1 --2- 172.21.15.67:0/481106964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8f9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.427+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/481106964 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8f5c10 msgr2=0x55b63a8f5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:45.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.427+0000 7f8a4f7fe700 1 --2- 172.21.15.67:0/481106964 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8f5c10 0x55b63a8f5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.427+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/481106964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 msgr2=0x55b63a8f9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:45.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.427+0000 7f8a4f7fe700 1 --2- 172.21.15.67:0/481106964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8f9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.427+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/481106964 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b63a795880 con 0x55b63a8fa230 2024-05-03T17:23:45.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.427+0000 7f8a4effd700 1 --2- 172.21.15.67:0/481106964 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8f5c10 0x55b63a8f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:23:45.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.427+0000 7f8a4f7fe700 1 --2- 172.21.15.67:0/481106964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 0x55b63a8fa690 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8a40004800 tx=0x7f8a400a9240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f068801c77c76679 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:45.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.427+0000 7f8a45ffb700 1 -- 172.21.15.67:0/481106964 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a400ada10 con 0x55b63a8fa230 2024-05-03T17:23:45.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.427+0000 7f8a45ffb700 1 -- 172.21.15.67:0/481106964 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a400b24c0 con 0x55b63a8fa230 2024-05-03T17:23:45.429 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.427+0000 7f8a45ffb700 1 -- 172.21.15.67:0/481106964 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4008fbd0 con 0x55b63a8fa230 2024-05-03T17:23:45.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.428+0000 7f8a6013db80 1 -- 172.21.15.67:0/481106964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 msgr2=0x55b63a8fa690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:45.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.428+0000 7f8a6013db80 1 --2- 172.21.15.67:0/481106964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 0x55b63a8fa690 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f8a40004800 tx=0x7f8a400a9240 comp rx=0 tx=0).stop 2024-05-03T17:23:45.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.428+0000 7f8a6013db80 1 -- 172.21.15.67:0/481106964 shutdown_connections 2024-05-03T17:23:45.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.428+0000 7f8a6013db80 1 --2- 172.21.15.67:0/481106964 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 0x55b63a8fa690 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.428+0000 7f8a6013db80 1 --2- 172.21.15.67:0/481106964 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8f5c10 0x55b63a8f5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.428+0000 7f8a6013db80 1 --2- 172.21.15.67:0/481106964 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8f9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.428+0000 7f8a6013db80 1 -- 172.21.15.67:0/481106964 >> 172.21.15.67:0/481106964 conn(0x55b63a8220b0 msgr2=0x55b63a8f0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:45.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.428+0000 7f8a6013db80 1 -- 172.21.15.67:0/481106964 shutdown_connections 2024-05-03T17:23:45.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.428+0000 7f8a6013db80 1 -- 172.21.15.67:0/481106964 wait complete. 2024-05-03T17:23:45.433 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.428+0000 7f8a6013db80 1 -- start start 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.428+0000 7f8a6013db80 1 Processor -- start 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a6013db80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f5c10 0x55b63a9016f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a6013db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a6013db80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8fa230 0x55b63a8fefd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a6013db80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b63a902490 con 0x55b63a8f5c10 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a6013db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b63a7aebe0 con 0x55b63a8f6530 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a6013db80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b63a902610 con 0x55b63a8fa230 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a6013db80 10 client.0 _ll_register_callbacks cb 0x55b63a7e08d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a4effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f5c10 0x55b63a9016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a4f7fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8fa230 0x55b63a8fefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a4effd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f5c10 0x55b63a9016f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48590/0 (socket says 172.21.15.67:48590) 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a4effd700 1 -- 172.21.15.67:0/3695014189 learned_addr learned my addr 172.21.15.67:0/3695014189 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a4e7fc700 1 --2- 172.21.15.67:0/3695014189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.434 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a4effd700 1 -- 172.21.15.67:0/3695014189 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8fa230 msgr2=0x55b63a8fefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a4effd700 1 --2- 172.21.15.67:0/3695014189 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8fa230 0x55b63a8fefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a4effd700 1 -- 172.21.15.67:0/3695014189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 msgr2=0x55b63a8fea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a4effd700 1 --2- 172.21.15.67:0/3695014189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8fea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a4effd700 1 -- 172.21.15.67:0/3695014189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b63a7e4640 con 0x55b63a8f5c10 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a4effd700 1 --2- 172.21.15.67:0/3695014189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f5c10 0x55b63a9016f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a480166a0 tx=0x7f8a48004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17f60941d40fdc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a2effd700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a2effd700 1 -- 172.21.15.67:0/3695014189 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a48016e40 con 0x55b63a8f5c10 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a2effd700 1 -- 172.21.15.67:0/3695014189 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f8a480215c0 con 0x55b63a8f5c10 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.429+0000 7f8a6013db80 1 -- 172.21.15.67:0/3695014189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a400b24c0 con 0x55b63a8f5c10 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.430+0000 7f8a2effd700 1 -- 172.21.15.67:0/3695014189 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8a480217c0 con 0x55b63a8f5c10 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.430+0000 7f8a6013db80 10 client.15063 fetch_fsmap learned FSMap version 25 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.430+0000 7f8a6013db80 1 -- 172.21.15.67:0/3695014189 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b63a90cdc0 con 0x55b63a8f5c10 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.430+0000 7f8a2effd700 1 -- 172.21.15.67:0/3695014189 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a400b24c0 con 0x55b63a8f5c10 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.430+0000 7f8a2effd700 1 -- 172.21.15.67:0/3695014189 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f8a48022a90 con 0x55b63a8f5c10 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.430+0000 7f8a6013db80 10 client.15063 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T17:23:45.435 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.430+0000 7f8a6013db80 10 client.15063 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.430+0000 7f8a6013db80 20 client.15063 populate_metadata read hostname 'smithi067' 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.430+0000 7f8a6013db80 1 --2- 172.21.15.67:0/3695014189 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x55b63a90f410 0x55b63a910880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.430+0000 7f8a6013db80 4 client.15063 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.430+0000 7f8a6013db80 1 -- 172.21.15.67:0/3695014189 --> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b63a8561f0 con 0x55b63a90f410 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.430+0000 7f8a4e7fc700 1 --2- 172.21.15.67:0/3695014189 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x55b63a90f410 0x55b63a910880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.431+0000 7f8a4e7fc700 1 --2- 172.21.15.67:0/3695014189 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x55b63a90f410 0x55b63a910880 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bda0a80c8671c8d0 server_cookie=7317e951814fe45c in_seq=0 out_seq=0 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.431+0000 7f8a2effd700 10 client.15063 ms_handle_connect on v2:172.21.15.70:6836/3323713018 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.431+0000 7f8a2effd700 1 -- 172.21.15.67:0/3695014189 <== mds.1 v2:172.21.15.70:6836/3323713018 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b63a8561f0 con 0x55b63a90f410 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.431+0000 7f8a2effd700 10 client.15063 handle_command_reply: tid=0 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.431+0000 7f8a6013db80 1 client.15063 shutdown 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.432+0000 7f8a6013db80 1 -- 172.21.15.67:0/3695014189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f5c10 msgr2=0x55b63a9016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.432+0000 7f8a6013db80 1 --2- 172.21.15.67:0/3695014189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f5c10 0x55b63a9016f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a480166a0 tx=0x7f8a48004ad0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.432+0000 7f8a6013db80 1 -- 172.21.15.67:0/3695014189 shutdown_connections 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.432+0000 7f8a6013db80 1 --2- 172.21.15.67:0/3695014189 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8fa230 0x55b63a8fefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.436 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.432+0000 7f8a6013db80 1 --2- 172.21.15.67:0/3695014189 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x55b63a90f410 0x55b63a910880 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.432+0000 7f8a6013db80 1 --2- 172.21.15.67:0/3695014189 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f5c10 0x55b63a9016f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.432+0000 7f8a6013db80 1 --2- 172.21.15.67:0/3695014189 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8fea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.432+0000 7f8a6013db80 1 -- 172.21.15.67:0/3695014189 >> 172.21.15.67:0/3695014189 conn(0x55b63a8220b0 msgr2=0x55b63a8fb6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:45.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.432+0000 7f8a2effd700 0 client.15063 ms_handle_reset on v2:172.21.15.70:6836/3323713018 2024-05-03T17:23:45.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.432+0000 7f8a6013db80 1 -- 172.21.15.67:0/3695014189 shutdown_connections 2024-05-03T17:23:45.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.432+0000 7f8a6013db80 1 -- 172.21.15.67:0/3695014189 wait complete. 2024-05-03T17:23:45.437 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.432+0000 7f8a6013db80 20 client.15063 trim_cache size 0 max 16384 2024-05-03T17:23:45.454 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a6013db80 1 Processor -- start 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a6013db80 1 -- start start 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a6013db80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8f5c10 0x55b63a8ff000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a6013db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8f5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a6013db80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 0x55b63a8f7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a6013db80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b63a90ebc0 con 0x55b63a8fa230 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a6013db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b63a90f290 con 0x55b63a8f6530 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a6013db80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b63a90f960 con 0x55b63a8f5c10 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a4f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 0x55b63a8f7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a4f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 0x55b63a8f7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48592/0 (socket says 172.21.15.67:48592) 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/1133331144 learned_addr learned my addr 172.21.15.67:0/1133331144 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a4effd700 1 --2- 172.21.15.67:0/1133331144 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8f5c10 0x55b63a8ff000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.453+0000 7f8a4e7fc700 1 --2- 172.21.15.67:0/1133331144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.454+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/1133331144 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8f5c10 msgr2=0x55b63a8ff000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:45.455 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.454+0000 7f8a4f7fe700 1 --2- 172.21.15.67:0/1133331144 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8f5c10 0x55b63a8ff000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.454+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/1133331144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 msgr2=0x55b63a8f5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:45.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.454+0000 7f8a4f7fe700 1 --2- 172.21.15.67:0/1133331144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8f5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.454+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/1133331144 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b63a9025c0 con 0x55b63a8fa230 2024-05-03T17:23:45.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.454+0000 7f8a4e7fc700 1 --2- 172.21.15.67:0/1133331144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:23:45.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.454+0000 7f8a4f7fe700 1 --2- 172.21.15.67:0/1133331144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 0x55b63a8f7c60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8a40009fd0 tx=0x7f8a400cc040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a74bc2142059e1a server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:45.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.454+0000 7f8a457fa700 1 -- 172.21.15.67:0/1133331144 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a400a5520 con 0x55b63a8fa230 2024-05-03T17:23:45.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.454+0000 7f8a457fa700 1 -- 172.21.15.67:0/1133331144 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a400b24c0 con 0x55b63a8fa230 2024-05-03T17:23:45.456 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.454+0000 7f8a457fa700 1 -- 172.21.15.67:0/1133331144 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a400a98c0 con 0x55b63a8fa230 2024-05-03T17:23:45.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.455+0000 7f8a6013db80 1 -- 172.21.15.67:0/1133331144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 msgr2=0x55b63a8f7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:45.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.455+0000 7f8a6013db80 1 --2- 172.21.15.67:0/1133331144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 0x55b63a8f7c60 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8a40009fd0 tx=0x7f8a400cc040 comp rx=0 tx=0).stop 2024-05-03T17:23:45.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.455+0000 7f8a6013db80 1 -- 172.21.15.67:0/1133331144 shutdown_connections 2024-05-03T17:23:45.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.455+0000 7f8a6013db80 1 --2- 172.21.15.67:0/1133331144 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8fa230 0x55b63a8f7c60 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.455+0000 7f8a6013db80 1 --2- 172.21.15.67:0/1133331144 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8f5c10 0x55b63a8ff000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.455+0000 7f8a6013db80 1 --2- 172.21.15.67:0/1133331144 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f6530 0x55b63a8f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.455+0000 7f8a6013db80 1 -- 172.21.15.67:0/1133331144 >> 172.21.15.67:0/1133331144 conn(0x55b63a8220b0 msgr2=0x55b63a8f4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:45.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.455+0000 7f8a6013db80 1 -- 172.21.15.67:0/1133331144 shutdown_connections 2024-05-03T17:23:45.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.455+0000 7f8a6013db80 1 -- 172.21.15.67:0/1133331144 wait complete. 2024-05-03T17:23:45.458 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.455+0000 7f8a6013db80 1 -- start start 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.455+0000 7f8a6013db80 1 Processor -- start 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a6013db80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f5c10 0x55b63a91a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a6013db80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f6530 0x55b63a8f72d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a6013db80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8fa230 0x55b63a8f7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a6013db80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55b63a910700 con 0x55b63a8f6530 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a6013db80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55b63a910030 con 0x55b63a8f5c10 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a6013db80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55b63a910dd0 con 0x55b63a8fa230 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a6013db80 10 client.0 _ll_register_callbacks cb 0x55b63a7e08d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a4e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f6530 0x55b63a8f72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a4f7fe700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8fa230 0x55b63a8f7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a4effd700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f5c10 0x55b63a91a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a4e7fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f6530 0x55b63a8f72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:48606/0 (socket says 172.21.15.67:48606) 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a4e7fc700 1 -- 172.21.15.67:0/3516098797 learned_addr learned my addr 172.21.15.67:0/3516098797 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:45.459 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/3516098797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f5c10 msgr2=0x55b63a91a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:45.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a4f7fe700 1 --2- 172.21.15.67:0/3516098797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f5c10 0x55b63a91a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/3516098797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f6530 msgr2=0x55b63a8f72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:45.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a4f7fe700 1 --2- 172.21.15.67:0/3516098797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f6530 0x55b63a8f72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a4f7fe700 1 -- 172.21.15.67:0/3516098797 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b63a8f8000 con 0x55b63a8fa230 2024-05-03T17:23:45.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a4e7fc700 1 --2- 172.21.15.67:0/3516098797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f6530 0x55b63a8f72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:23:45.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a4f7fe700 1 --2- 172.21.15.67:0/3516098797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8fa230 0x55b63a8f7810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55b63a8ffcb0 tx=0x7f8a40005700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=189604cb5fd1e032 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:45.460 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.456+0000 7f8a2effd700 10 client.0 ms_handle_connect on v2:172.21.15.82:3300/0 2024-05-03T17:23:45.462 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.458+0000 7f8a2effd700 1 -- 172.21.15.67:0/3516098797 <== mon.1 v2:172.21.15.82:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a400a9430 con 0x55b63a8fa230 2024-05-03T17:23:45.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.458+0000 7f8a2effd700 1 -- 172.21.15.67:0/3516098797 <== mon.1 v2:172.21.15.82:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f8a400a98c0 con 0x55b63a8fa230 2024-05-03T17:23:45.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.458+0000 7f8a6013db80 1 -- 172.21.15.67:0/3516098797 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a400b24c0 con 0x55b63a8fa230 2024-05-03T17:23:45.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.459+0000 7f8a2effd700 1 -- 172.21.15.67:0/3516098797 <== mon.1 v2:172.21.15.82:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8a400ada10 con 0x55b63a8fa230 2024-05-03T17:23:45.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.459+0000 7f8a6013db80 10 client.24874 fetch_fsmap learned FSMap version 25 2024-05-03T17:23:45.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.459+0000 7f8a6013db80 1 -- 172.21.15.67:0/3516098797 --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b63a917c10 con 0x55b63a8fa230 2024-05-03T17:23:45.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.459+0000 7f8a2effd700 1 -- 172.21.15.67:0/3516098797 <== mon.1 v2:172.21.15.82:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a400b24c0 con 0x55b63a8fa230 2024-05-03T17:23:45.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.459+0000 7f8a2effd700 1 -- 172.21.15.67:0/3516098797 <== mon.1 v2:172.21.15.82:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7f8a400a88c0 con 0x55b63a8fa230 2024-05-03T17:23:45.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.459+0000 7f8a6013db80 10 client.24874 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T17:23:45.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.459+0000 7f8a6013db80 10 client.24874 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-03T17:23:45.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.459+0000 7f8a6013db80 20 client.24874 populate_metadata read hostname 'smithi067' 2024-05-03T17:23:45.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.459+0000 7f8a6013db80 1 --2- 172.21.15.67:0/3516098797 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x55b63a97eb20 0x55b63a97d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:45.463 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.459+0000 7f8a6013db80 4 client.24874 mds_command: new command op to 24446 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.460+0000 7f8a6013db80 1 -- 172.21.15.67:0/3516098797 --> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] -- command(tid 0: {"prefix": "status"}) -- 0x55b63a850290 con 0x55b63a97eb20 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.460+0000 7f8a4effd700 1 --2- 172.21.15.67:0/3516098797 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x55b63a97eb20 0x55b63a97d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.460+0000 7f8a2effd700 10 client.24874 ms_handle_connect on v2:172.21.15.70:6836/3323713018 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.460+0000 7f8a4effd700 1 --2- 172.21.15.67:0/3516098797 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x55b63a97eb20 0x55b63a97d830 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=21be9335466b523b server_cookie=820c4eca63e27554 in_seq=0 out_seq=0 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.460+0000 7f8a2effd700 1 -- 172.21.15.67:0/3516098797 <== mds.1 v2:172.21.15.70:6836/3323713018 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55b63a850290 con 0x55b63a97eb20 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.460+0000 7f8a2effd700 10 client.24874 handle_command_reply: tid=0 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.460+0000 7f8a6013db80 1 client.24874 shutdown 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.460+0000 7f8a6013db80 1 -- 172.21.15.67:0/3516098797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8fa230 msgr2=0x55b63a8f7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a6013db80 1 --2- 172.21.15.67:0/3516098797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8fa230 0x55b63a8f7810 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x55b63a8ffcb0 tx=0x7f8a40005700 comp rx=0 tx=0).stop 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a6013db80 1 -- 172.21.15.67:0/3516098797 shutdown_connections 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a6013db80 1 --2- 172.21.15.67:0/3516098797 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55b63a8fa230 0x55b63a8f7810 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a6013db80 1 --2- 172.21.15.67:0/3516098797 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55b63a8f5c10 0x55b63a91a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a6013db80 1 --2- 172.21.15.67:0/3516098797 >> [v2:172.21.15.70:6836/3323713018,v1:172.21.15.70:6837/3323713018] conn(0x55b63a97eb20 0x55b63a97d830 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a6013db80 1 --2- 172.21.15.67:0/3516098797 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55b63a8f6530 0x55b63a8f72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.464 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a2effd700 0 client.24874 ms_handle_reset on v2:172.21.15.70:6836/3323713018 2024-05-03T17:23:45.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a6013db80 1 -- 172.21.15.67:0/3516098797 >> 172.21.15.67:0/3516098797 conn(0x55b63a8220b0 msgr2=0x55b63a8f52b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:45.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a6013db80 1 -- 172.21.15.67:0/3516098797 shutdown_connections 2024-05-03T17:23:45.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a6013db80 1 -- 172.21.15.67:0/3516098797 wait complete. 2024-05-03T17:23:45.465 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a6013db80 20 client.24874 trim_cache size 0 max 16384 2024-05-03T17:23:45.465 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T17:23:45.465 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "20c8e22e-0970-11ef-bc97-c7b262605968", 2024-05-03T17:23:45.465 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 1, 2024-05-03T17:23:45.465 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24446, 2024-05-03T17:23:45.465 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-03T17:23:45.465 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-03T17:23:45.466 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-03T17:23:45.466 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 170.91888996099999, 2024-05-03T17:23:45.466 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-03T17:23:45.466 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 81, 2024-05-03T17:23:45.466 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 79, 2024-05-03T17:23:45.466 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 215.769933888 2024-05-03T17:23:45.466 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T17:23:45.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a558aa700 1 -- 172.21.15.67:0/1394140693 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8a3c0b2990 msgr2=0x7f8a3c0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:45.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a558aa700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8a3c0b2990 0x7f8a3c0b4e50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f8a5019ab30 tx=0x7f8a4800b040 comp rx=0 tx=0).stop 2024-05-03T17:23:45.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a558aa700 1 -- 172.21.15.67:0/1394140693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a50109500 msgr2=0x7f8a501999f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:45.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.461+0000 7f8a558aa700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a50109500 0x7f8a501999f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f8a4000d980 tx=0x7f8a4000dc90 comp rx=0 tx=0).stop 2024-05-03T17:23:45.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.462+0000 7f8a558aa700 1 -- 172.21.15.67:0/1394140693 shutdown_connections 2024-05-03T17:23:45.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.462+0000 7f8a558aa700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f8a3c0b2990 0x7f8a3c0b4e50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.462+0000 7f8a558aa700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8a50109500 0x7f8a501999f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.462+0000 7f8a558aa700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f8a5006ba10 0x7f8a5019f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.462+0000 7f8a558aa700 1 --2- 172.21.15.67:0/1394140693 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f8a5006c3d0 0x7f8a5019fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:45.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.462+0000 7f8a558aa700 1 -- 172.21.15.67:0/1394140693 >> 172.21.15.67:0/1394140693 conn(0x7f8a50079310 msgr2=0x7f8a501011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:45.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.462+0000 7f8a558aa700 1 -- 172.21.15.67:0/1394140693 shutdown_connections 2024-05-03T17:23:45.467 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:45.462+0000 7f8a558aa700 1 -- 172.21.15.67:0/1394140693 wait complete. 2024-05-03T17:23:46.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:45 smithi082 ceph-mon[33535]: pgmap v390: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:46.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:45 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:23:46.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:45 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:23:46.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:45 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:46.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:45 smithi070 ceph-mon[33250]: pgmap v390: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:46.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:45 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:23:46.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:45 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:23:46.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:45 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:46.132 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:45 smithi067 ceph-mon[29731]: pgmap v390: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:23:46.132 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:45 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:23:46.132 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:45 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:23:46.132 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:45 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:23:46.166 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph tell mds.b status 2024-05-03T17:23:46.420 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:47.928 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:47 smithi067 ceph-mon[29731]: pgmap v391: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:48.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:47 smithi082 ceph-mon[33535]: pgmap v391: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:48.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:47 smithi070 ceph-mon[33250]: pgmap v391: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:23:48.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.178+0000 7fd03742b700 1 -- 172.21.15.67:0/291333316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd03010ff00 msgr2=0x7fd0301122f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:48.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.178+0000 7fd03742b700 1 --2- 172.21.15.67:0/291333316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd03010ff00 0x7fd0301122f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fd028009990 tx=0x7fd028009ca0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.178+0000 7fd03742b700 1 -- 172.21.15.67:0/291333316 shutdown_connections 2024-05-03T17:23:48.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.178+0000 7fd03742b700 1 --2- 172.21.15.67:0/291333316 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd03010ff00 0x7fd0301122f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.178+0000 7fd03742b700 1 --2- 172.21.15.67:0/291333316 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd0300756d0 0x7fd030075ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.178+0000 7fd03742b700 1 --2- 172.21.15.67:0/291333316 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd030075ff0 0x7fd03010f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.178+0000 7fd03742b700 1 -- 172.21.15.67:0/291333316 >> 172.21.15.67:0/291333316 conn(0x7fd03007a530 msgr2=0x7fd03007a940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:48.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.178+0000 7fd03742b700 1 -- 172.21.15.67:0/291333316 shutdown_connections 2024-05-03T17:23:48.180 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.178+0000 7fd03742b700 1 -- 172.21.15.67:0/291333316 wait complete. 2024-05-03T17:23:48.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.179+0000 7fd03742b700 1 Processor -- start 2024-05-03T17:23:48.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.179+0000 7fd03742b700 1 -- start start 2024-05-03T17:23:48.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.179+0000 7fd03742b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0300756d0 0x7fd0301a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.179+0000 7fd03742b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd030075ff0 0x7fd0301a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.179+0000 7fd03742b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd03010ff00 0x7fd03019ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.179+0000 7fd03742b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fd030115860 con 0x7fd030075ff0 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.179+0000 7fd03742b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fd0301156e0 con 0x7fd03010ff00 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.179+0000 7fd03742b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fd0301159e0 con 0x7fd0300756d0 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.179+0000 7fd0349c6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd030075ff0 0x7fd0301a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.179+0000 7fd0349c6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd030075ff0 0x7fd0301a1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44098/0 (socket says 172.21.15.67:44098) 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.179+0000 7fd0349c6700 1 -- 172.21.15.67:0/574688819 learned_addr learned my addr 172.21.15.67:0/574688819 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.179+0000 7fd0359c8700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd03010ff00 0x7fd03019ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.180+0000 7fd0351c7700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0300756d0 0x7fd0301a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.180+0000 7fd0349c6700 1 -- 172.21.15.67:0/574688819 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0300756d0 msgr2=0x7fd0301a18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.180+0000 7fd0349c6700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0300756d0 0x7fd0301a18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.180+0000 7fd0349c6700 1 -- 172.21.15.67:0/574688819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd03010ff00 msgr2=0x7fd03019ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.180+0000 7fd0349c6700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd03010ff00 0x7fd03019ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.180+0000 7fd0349c6700 1 -- 172.21.15.67:0/574688819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03019c1e0 con 0x7fd030075ff0 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.180+0000 7fd0349c6700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd030075ff0 0x7fd0301a1e00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd02400c9b0 tx=0x7fd02400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.180+0000 7fd02e7fc700 1 -- 172.21.15.67:0/574688819 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd024007b00 con 0x7fd030075ff0 2024-05-03T17:23:48.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.180+0000 7fd03742b700 1 -- 172.21.15.67:0/574688819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd03019c4d0 con 0x7fd030075ff0 2024-05-03T17:23:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.180+0000 7fd02e7fc700 1 -- 172.21.15.67:0/574688819 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd02400ce50 con 0x7fd030075ff0 2024-05-03T17:23:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.180+0000 7fd02e7fc700 1 -- 172.21.15.67:0/574688819 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd024018660 con 0x7fd030075ff0 2024-05-03T17:23:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.180+0000 7fd03742b700 1 -- 172.21.15.67:0/574688819 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0301a8790 con 0x7fd030075ff0 2024-05-03T17:23:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.183+0000 7fd02e7fc700 1 -- 172.21.15.67:0/574688819 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fd02400f450 con 0x7fd030075ff0 2024-05-03T17:23:48.184 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.183+0000 7fd02e7fc700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0180b2910 0x7fd0180b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.183+0000 7fd0351c7700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0180b2910 0x7fd0180b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.183+0000 7fd02e7fc700 1 -- 172.21.15.67:0/574688819 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fd0240e2250 con 0x7fd030075ff0 2024-05-03T17:23:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.184+0000 7fd0351c7700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0180b2910 0x7fd0180b4dd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd0200060b0 tx=0x7fd02000c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.184+0000 7fd041cbeb80 1 Processor -- start 2024-05-03T17:23:48.185 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.184+0000 7fd041cbeb80 1 -- start start 2024-05-03T17:23:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.184+0000 7fd041cbeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be13740 0x55fe3be13b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.184+0000 7fd041cbeb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be14060 0x55fe3be0ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.184+0000 7fd041cbeb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be0f480 0x55fe3be0f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.187 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.184+0000 7fd041cbeb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe3bd618f0 con 0x55fe3be0f480 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.184+0000 7fd041cbeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fe3bcc6d50 con 0x55fe3be13740 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.184+0000 7fd041cbeb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fe3bd5d8b0 con 0x55fe3be14060 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0359c8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be0f480 0x55fe3be0f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0359c8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be0f480 0x55fe3be0f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44100/0 (socket says 172.21.15.67:44100) 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0359c8700 1 -- 172.21.15.67:0/1898644691 learned_addr learned my addr 172.21.15.67:0/1898644691 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0349c6700 1 --2- 172.21.15.67:0/1898644691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be14060 0x55fe3be0ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0351c7700 1 --2- 172.21.15.67:0/1898644691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be13740 0x55fe3be13b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0359c8700 1 -- 172.21.15.67:0/1898644691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be14060 msgr2=0x55fe3be0ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0359c8700 1 --2- 172.21.15.67:0/1898644691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be14060 0x55fe3be0ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0359c8700 1 -- 172.21.15.67:0/1898644691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be13740 msgr2=0x55fe3be13b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0359c8700 1 --2- 172.21.15.67:0/1898644691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be13740 0x55fe3be13b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0359c8700 1 -- 172.21.15.67:0/1898644691 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe3bcae880 con 0x55fe3be0f480 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0359c8700 1 --2- 172.21.15.67:0/1898644691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be0f480 0x55fe3be0f8c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd028015000 tx=0x7fd028009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56c66e819fe9958a server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0177fe700 1 -- 172.21.15.67:0/1898644691 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02800bad0 con 0x55fe3be0f480 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0177fe700 1 -- 172.21.15.67:0/1898644691 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd028026070 con 0x55fe3be0f480 2024-05-03T17:23:48.188 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.185+0000 7fd0177fe700 1 -- 172.21.15.67:0/1898644691 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0280054c0 con 0x55fe3be0f480 2024-05-03T17:23:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.186+0000 7fd041cbeb80 1 -- 172.21.15.67:0/1898644691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be0f480 msgr2=0x55fe3be0f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.186+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/1898644691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be0f480 0x55fe3be0f8c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd028015000 tx=0x7fd028009f90 comp rx=0 tx=0).stop 2024-05-03T17:23:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.186+0000 7fd041cbeb80 1 -- 172.21.15.67:0/1898644691 shutdown_connections 2024-05-03T17:23:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.186+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/1898644691 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be0f480 0x55fe3be0f8c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.186+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/1898644691 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be13740 0x55fe3be13b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.186+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/1898644691 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be14060 0x55fe3be0ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.186+0000 7fd041cbeb80 1 -- 172.21.15.67:0/1898644691 >> 172.21.15.67:0/1898644691 conn(0x55fe3bd3bb20 msgr2=0x55fe3be099a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.186+0000 7fd041cbeb80 1 -- 172.21.15.67:0/1898644691 shutdown_connections 2024-05-03T17:23:48.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.186+0000 7fd041cbeb80 1 -- 172.21.15.67:0/1898644691 wait complete. 2024-05-03T17:23:48.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.186+0000 7fd041cbeb80 1 -- start start 2024-05-03T17:23:48.192 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.186+0000 7fd041cbeb80 1 Processor -- start 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd041cbeb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be0f480 0x55fe3be1acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd041cbeb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 0x55fe3be18090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd041cbeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be14060 0x55fe3be185d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd041cbeb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe3be1bb40 con 0x55fe3be13740 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd041cbeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fe3bd1ae80 con 0x55fe3be14060 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd041cbeb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fe3be1bcc0 con 0x55fe3be0f480 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd041cbeb80 10 client.0 _ll_register_callbacks cb 0x55fe3bcf5560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd0349c6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 0x55fe3be18090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd0349c6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 0x55fe3be18090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44114/0 (socket says 172.21.15.67:44114) 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd0349c6700 1 -- 172.21.15.67:0/4076932845 learned_addr learned my addr 172.21.15.67:0/4076932845 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd0359c8700 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be14060 0x55fe3be185d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd0351c7700 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be0f480 0x55fe3be1acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd0349c6700 1 -- 172.21.15.67:0/4076932845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be0f480 msgr2=0x55fe3be1acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd0349c6700 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be0f480 0x55fe3be1acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.193 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd0349c6700 1 -- 172.21.15.67:0/4076932845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be14060 msgr2=0x55fe3be185d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd0349c6700 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be14060 0x55fe3be185d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd0349c6700 1 -- 172.21.15.67:0/4076932845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe3bd18710 con 0x55fe3be13740 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.187+0000 7fd0359c8700 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be14060 0x55fe3be185d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd0349c6700 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 0x55fe3be18090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd02400afd0 tx=0x7fd0240e0070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c398b90fb17562 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd014ff9700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd014ff9700 1 -- 172.21.15.67:0/4076932845 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02406a020 con 0x55fe3be13740 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd014ff9700 1 -- 172.21.15.67:0/4076932845 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fd024007ca0 con 0x55fe3be13740 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd041cbeb80 1 -- 172.21.15.67:0/4076932845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd028026070 con 0x55fe3be13740 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd014ff9700 1 -- 172.21.15.67:0/4076932845 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd0240a9e00 con 0x55fe3be13740 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd041cbeb80 10 client.15081 fetch_fsmap learned FSMap version 25 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd041cbeb80 1 -- 172.21.15.67:0/4076932845 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fe3be262c0 con 0x55fe3be13740 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd014ff9700 1 -- 172.21.15.67:0/4076932845 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd028026070 con 0x55fe3be13740 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd014ff9700 1 -- 172.21.15.67:0/4076932845 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fd02406d020 con 0x55fe3be13740 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd041cbeb80 10 client.15081 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T17:23:48.194 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd041cbeb80 10 client.15081 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd041cbeb80 20 client.15081 populate_metadata read hostname 'smithi067' 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x55fe3be28900 0x55fe3be29d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd041cbeb80 4 client.15081 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.188+0000 7fd041cbeb80 1 -- 172.21.15.67:0/4076932845 --> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe3bd6f2d0 con 0x55fe3be28900 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.189+0000 7fd0351c7700 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x55fe3be28900 0x55fe3be29d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.189+0000 7fd014ff9700 10 client.15081 ms_handle_connect on v2:172.21.15.70:6838/902276547 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.189+0000 7fd0351c7700 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x55fe3be28900 0x55fe3be29d70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=183883bd6bc067c6 server_cookie=68c12ed422cfb8fa in_seq=0 out_seq=0 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.190+0000 7fd014ff9700 1 -- 172.21.15.67:0/4076932845 <== mds.0 v2:172.21.15.70:6838/902276547 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fe3bd6f2d0 con 0x55fe3be28900 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.190+0000 7fd014ff9700 10 client.15081 handle_command_reply: tid=0 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.190+0000 7fd041cbeb80 1 client.15081 shutdown 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.190+0000 7fd041cbeb80 1 -- 172.21.15.67:0/4076932845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 msgr2=0x55fe3be18090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.190+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 0x55fe3be18090 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd02400afd0 tx=0x7fd0240e0070 comp rx=0 tx=0).stop 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.190+0000 7fd041cbeb80 1 -- 172.21.15.67:0/4076932845 shutdown_connections 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.190+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be14060 0x55fe3be185d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.190+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x55fe3be28900 0x55fe3be29d70 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.190+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be0f480 0x55fe3be1acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.195 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.190+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/4076932845 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 0x55fe3be18090 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.190+0000 7fd041cbeb80 1 -- 172.21.15.67:0/4076932845 >> 172.21.15.67:0/4076932845 conn(0x55fe3bd3bb20 msgr2=0x55fe3be14d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:48.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.190+0000 7fd014ff9700 0 client.15081 ms_handle_reset on v2:172.21.15.70:6838/902276547 2024-05-03T17:23:48.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.191+0000 7fd041cbeb80 1 -- 172.21.15.67:0/4076932845 shutdown_connections 2024-05-03T17:23:48.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.191+0000 7fd041cbeb80 1 -- 172.21.15.67:0/4076932845 wait complete. 2024-05-03T17:23:48.196 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.191+0000 7fd041cbeb80 20 client.15081 trim_cache size 0 max 16384 2024-05-03T17:23:48.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd041cbeb80 1 Processor -- start 2024-05-03T17:23:48.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd041cbeb80 1 -- start start 2024-05-03T17:23:48.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd041cbeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be0f480 0x55fe3be09810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.214 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd041cbeb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be13740 0x55fe3be15510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd041cbeb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be14060 0x55fe3be15a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd041cbeb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe3be280b0 con 0x55fe3be14060 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd041cbeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fe3be28780 con 0x55fe3be0f480 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd041cbeb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fe3be28e50 con 0x55fe3be13740 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd0359c8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be14060 0x55fe3be15a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd0359c8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be14060 0x55fe3be15a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44122/0 (socket says 172.21.15.67:44122) 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd0359c8700 1 -- 172.21.15.67:0/3440440423 learned_addr learned my addr 172.21.15.67:0/3440440423 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd0349c6700 1 --2- 172.21.15.67:0/3440440423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be13740 0x55fe3be15510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd0351c7700 1 --2- 172.21.15.67:0/3440440423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be0f480 0x55fe3be09810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd0359c8700 1 -- 172.21.15.67:0/3440440423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be13740 msgr2=0x55fe3be15510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd0359c8700 1 --2- 172.21.15.67:0/3440440423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be13740 0x55fe3be15510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd0359c8700 1 -- 172.21.15.67:0/3440440423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be0f480 msgr2=0x55fe3be09810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd0359c8700 1 --2- 172.21.15.67:0/3440440423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be0f480 0x55fe3be09810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd0359c8700 1 -- 172.21.15.67:0/3440440423 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe3be0b800 con 0x55fe3be14060 2024-05-03T17:23:48.215 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.212+0000 7fd0351c7700 1 --2- 172.21.15.67:0/3440440423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be0f480 0x55fe3be09810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:23:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.213+0000 7fd0359c8700 1 --2- 172.21.15.67:0/3440440423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be14060 0x55fe3be15a50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd028006010 tx=0x7fd02800bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72fdbfe8168eef42 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.213+0000 7fd016ffd700 1 -- 172.21.15.67:0/3440440423 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd028005310 con 0x55fe3be14060 2024-05-03T17:23:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.213+0000 7fd016ffd700 1 -- 172.21.15.67:0/3440440423 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd028026030 con 0x55fe3be14060 2024-05-03T17:23:48.216 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.213+0000 7fd016ffd700 1 -- 172.21.15.67:0/3440440423 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02800f750 con 0x55fe3be14060 2024-05-03T17:23:48.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.213+0000 7fd041cbeb80 1 -- 172.21.15.67:0/3440440423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be14060 msgr2=0x55fe3be15a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.213+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/3440440423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be14060 0x55fe3be15a50 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd028006010 tx=0x7fd02800bd90 comp rx=0 tx=0).stop 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.213+0000 7fd041cbeb80 1 -- 172.21.15.67:0/3440440423 shutdown_connections 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.213+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/3440440423 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be14060 0x55fe3be15a50 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.213+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/3440440423 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be0f480 0x55fe3be09810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.213+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/3440440423 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be13740 0x55fe3be15510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.213+0000 7fd041cbeb80 1 -- 172.21.15.67:0/3440440423 >> 172.21.15.67:0/3440440423 conn(0x55fe3bd3bb20 msgr2=0x55fe3be169a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.214+0000 7fd041cbeb80 1 -- 172.21.15.67:0/3440440423 shutdown_connections 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.214+0000 7fd041cbeb80 1 -- 172.21.15.67:0/3440440423 wait complete. 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.214+0000 7fd041cbeb80 1 -- start start 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.214+0000 7fd041cbeb80 1 Processor -- start 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.214+0000 7fd041cbeb80 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be0f480 0x55fe3be15a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.214+0000 7fd041cbeb80 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 0x55fe3be331e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.214+0000 7fd041cbeb80 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be14060 0x55fe3be33720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.214+0000 7fd041cbeb80 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x55fe3be29bf0 con 0x55fe3be13740 2024-05-03T17:23:48.220 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.214+0000 7fd041cbeb80 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x55fe3be29520 con 0x55fe3be0f480 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.214+0000 7fd041cbeb80 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x55fe3be2a2c0 con 0x55fe3be14060 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.214+0000 7fd041cbeb80 10 client.0 _ll_register_callbacks cb 0x55fe3bcf5560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0349c6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 0x55fe3be331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0349c6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 0x55fe3be331e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44136/0 (socket says 172.21.15.67:44136) 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0349c6700 1 -- 172.21.15.67:0/100750137 learned_addr learned my addr 172.21.15.67:0/100750137 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0359c8700 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be14060 0x55fe3be33720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0351c7700 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be0f480 0x55fe3be15a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0349c6700 1 -- 172.21.15.67:0/100750137 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be14060 msgr2=0x55fe3be33720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0349c6700 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be14060 0x55fe3be33720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0349c6700 1 -- 172.21.15.67:0/100750137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be0f480 msgr2=0x55fe3be15a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0349c6700 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be0f480 0x55fe3be15a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0349c6700 1 -- 172.21.15.67:0/100750137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe3be0c5a0 con 0x55fe3be13740 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0359c8700 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be14060 0x55fe3be33720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0351c7700 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be0f480 0x55fe3be15a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:23:48.221 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd0349c6700 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 0x55fe3be331e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd0240a6970 tx=0x7fd0240bed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ffaba3e6bce099 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd014ff9700 10 client.0 ms_handle_connect on v2:172.21.15.67:3300/0 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd014ff9700 1 -- 172.21.15.67:0/100750137 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02406a070 con 0x55fe3be13740 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd014ff9700 1 -- 172.21.15.67:0/100750137 <== mon.0 v2:172.21.15.67:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fd024007ca0 con 0x55fe3be13740 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd041cbeb80 1 -- 172.21.15.67:0/100750137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd028026030 con 0x55fe3be13740 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd014ff9700 1 -- 172.21.15.67:0/100750137 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd0240a8e70 con 0x55fe3be13740 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd041cbeb80 10 client.15087 fetch_fsmap learned FSMap version 25 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd041cbeb80 1 -- 172.21.15.67:0/100750137 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55fe3be93ec0 con 0x55fe3be13740 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.215+0000 7fd014ff9700 1 -- 172.21.15.67:0/100750137 <== mon.0 v2:172.21.15.67:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd028026030 con 0x55fe3be13740 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.216+0000 7fd014ff9700 1 -- 172.21.15.67:0/100750137 <== mon.0 v2:172.21.15.67:3300/0 5 ==== fsmap(e 25) ==== 8338+0+0 (secure 0 0 0) 0x7fd02406d070 con 0x55fe3be13740 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.216+0000 7fd041cbeb80 10 client.15087 fetch_fsmap finished waiting for FSMap version 25 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.216+0000 7fd041cbeb80 10 client.15087 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.216+0000 7fd041cbeb80 20 client.15087 populate_metadata read hostname 'smithi067' 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.216+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x55fe3be98a30 0x55fe3be97740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.216+0000 7fd041cbeb80 4 client.15087 mds_command: new command op to 24464 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-03T17:23:48.222 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.216+0000 7fd041cbeb80 1 -- 172.21.15.67:0/100750137 --> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] -- command(tid 0: {"prefix": "status"}) -- 0x55fe3bd69370 con 0x55fe3be98a30 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.216+0000 7fd0351c7700 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x55fe3be98a30 0x55fe3be97740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.216+0000 7fd0351c7700 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x55fe3be98a30 0x55fe3be97740 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf8a7858399ee4bf server_cookie=b372ef6cafc0801f in_seq=0 out_seq=0 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.216+0000 7fd014ff9700 10 client.15087 ms_handle_connect on v2:172.21.15.70:6838/902276547 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd014ff9700 1 -- 172.21.15.67:0/100750137 <== mds.0 v2:172.21.15.70:6838/902276547 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fe3bd69370 con 0x55fe3be98a30 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd014ff9700 10 client.15087 handle_command_reply: tid=0 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd041cbeb80 1 client.15087 shutdown 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd041cbeb80 1 -- 172.21.15.67:0/100750137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 msgr2=0x55fe3be331e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 0x55fe3be331e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd0240a6970 tx=0x7fd0240bed80 comp rx=0 tx=0).stop 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd041cbeb80 1 -- 172.21.15.67:0/100750137 shutdown_connections 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.70:6838/902276547,v1:172.21.15.70:6839/902276547] conn(0x55fe3be98a30 0x55fe3be97740 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x55fe3be14060 0x55fe3be33720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x55fe3be0f480 0x55fe3be15a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd041cbeb80 1 --2- 172.21.15.67:0/100750137 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x55fe3be13740 0x55fe3be331e0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd014ff9700 0 client.15087 ms_handle_reset on v2:172.21.15.70:6838/902276547 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd041cbeb80 1 -- 172.21.15.67:0/100750137 >> 172.21.15.67:0/100750137 conn(0x55fe3bd3bb20 msgr2=0x55fe3be16e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd041cbeb80 1 -- 172.21.15.67:0/100750137 shutdown_connections 2024-05-03T17:23:48.223 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd041cbeb80 1 -- 172.21.15.67:0/100750137 wait complete. 2024-05-03T17:23:48.224 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd041cbeb80 20 client.15087 trim_cache size 0 max 16384 2024-05-03T17:23:48.224 INFO:teuthology.orchestra.run.smithi067.stdout:{ 2024-05-03T17:23:48.224 INFO:teuthology.orchestra.run.smithi067.stdout: "cluster_fsid": "20c8e22e-0970-11ef-bc97-c7b262605968", 2024-05-03T17:23:48.224 INFO:teuthology.orchestra.run.smithi067.stdout: "whoami": 0, 2024-05-03T17:23:48.224 INFO:teuthology.orchestra.run.smithi067.stdout: "id": 24464, 2024-05-03T17:23:48.224 INFO:teuthology.orchestra.run.smithi067.stdout: "want_state": "up:active", 2024-05-03T17:23:48.224 INFO:teuthology.orchestra.run.smithi067.stdout: "state": "up:active", 2024-05-03T17:23:48.224 INFO:teuthology.orchestra.run.smithi067.stdout: "fs_name": "cephfs", 2024-05-03T17:23:48.224 INFO:teuthology.orchestra.run.smithi067.stdout: "rank_uptime": 188.01325794900001, 2024-05-03T17:23:48.224 INFO:teuthology.orchestra.run.smithi067.stdout: "mdsmap_epoch": 25, 2024-05-03T17:23:48.224 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch": 81, 2024-05-03T17:23:48.225 INFO:teuthology.orchestra.run.smithi067.stdout: "osdmap_epoch_barrier": 79, 2024-05-03T17:23:48.225 INFO:teuthology.orchestra.run.smithi067.stdout: "uptime": 205.01562638799999 2024-05-03T17:23:48.225 INFO:teuthology.orchestra.run.smithi067.stdout:} 2024-05-03T17:23:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd03742b700 1 -- 172.21.15.67:0/574688819 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0180b2910 msgr2=0x7fd0180b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd03742b700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0180b2910 0x7fd0180b4dd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd0200060b0 tx=0x7fd02000c040 comp rx=0 tx=0).stop 2024-05-03T17:23:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd03742b700 1 -- 172.21.15.67:0/574688819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd030075ff0 msgr2=0x7fd0301a1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.217+0000 7fd03742b700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd030075ff0 0x7fd0301a1e00 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fd02400c9b0 tx=0x7fd02400ccc0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.218+0000 7fd03742b700 1 -- 172.21.15.67:0/574688819 shutdown_connections 2024-05-03T17:23:48.225 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.218+0000 7fd03742b700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fd0180b2910 0x7fd0180b4dd0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.218+0000 7fd03742b700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fd03010ff00 0x7fd03019ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.218+0000 7fd03742b700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fd0300756d0 0x7fd0301a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.218+0000 7fd03742b700 1 --2- 172.21.15.67:0/574688819 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd030075ff0 0x7fd0301a1e00 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:48.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.218+0000 7fd03742b700 1 -- 172.21.15.67:0/574688819 >> 172.21.15.67:0/574688819 conn(0x7fd03007a530 msgr2=0x7fd0301005f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:48.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.218+0000 7fd03742b700 1 -- 172.21.15.67:0/574688819 shutdown_connections 2024-05-03T17:23:48.226 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:48.218+0000 7fd03742b700 1 -- 172.21.15.67:0/574688819 wait complete. 2024-05-03T17:23:48.920 INFO:teuthology.run_tasks:Running task sequential... 2024-05-03T17:23:48.929 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-03T17:23:48.929 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-03T17:23:48.930 INFO:teuthology.task.print:Enabling mgr modules 2024-05-03T17:23:48.930 INFO:teuthology.run_tasks:Running task kclient... 2024-05-03T17:23:48.964 INFO:tasks.kclient:Mounting kernel clients... 2024-05-03T17:23:48.964 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-03T17:23:48.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:23:48.965 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-03T17:23:48.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:23:48.981 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-05-03T17:23:49.049 INFO:teuthology.orchestra.run.smithi067.stderr:Cannot find device "ceph-brx" 2024-05-03T17:23:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:23:49.051 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-05-03T17:23:49.051 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-03T17:23:49.051 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-03T17:23:49.052 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs ls 2024-05-03T17:23:49.339 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:23:50.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:49 smithi082 ceph-mon[33535]: pgmap v392: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:50.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:49 smithi070 ceph-mon[33250]: pgmap v392: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:50.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:49 smithi067 ceph-mon[29731]: pgmap v392: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:23:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.091+0000 7f533bed7700 1 -- 172.21.15.67:0/3049398071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5334108450 msgr2=0x7f5334108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.091+0000 7f533bed7700 1 --2- 172.21.15.67:0/3049398071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5334108450 0x7f5334108830 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5330009990 tx=0x7f5330009ca0 comp rx=0 tx=0).stop 2024-05-03T17:23:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.091+0000 7f533bed7700 1 -- 172.21.15.67:0/3049398071 shutdown_connections 2024-05-03T17:23:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.091+0000 7f533bed7700 1 --2- 172.21.15.67:0/3049398071 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5334102d90 0x7f533410f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.091+0000 7f533bed7700 1 --2- 172.21.15.67:0/3049398071 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5334108450 0x7f5334108830 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:51.095 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.091+0000 7f533bed7700 1 --2- 172.21.15.67:0/3049398071 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f53341023d0 0x7f5334102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.091+0000 7f533bed7700 1 -- 172.21.15.67:0/3049398071 >> 172.21.15.67:0/3049398071 conn(0x7f5334078410 msgr2=0x7f5334078820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.092+0000 7f533bed7700 1 -- 172.21.15.67:0/3049398071 shutdown_connections 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.092+0000 7f533bed7700 1 -- 172.21.15.67:0/3049398071 wait complete. 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.092+0000 7f533bed7700 1 Processor -- start 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.092+0000 7f533bed7700 1 -- start start 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.092+0000 7f533bed7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53341023d0 0x7f533419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.092+0000 7f533bed7700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5334102d90 0x7f533419fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.092+0000 7f533bed7700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5334108450 0x7f53341999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.092+0000 7f533bed7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f53341125a0 con 0x7f53341023d0 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.092+0000 7f533bed7700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f5334112420 con 0x7f5334108450 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.092+0000 7f533bed7700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f5334112720 con 0x7f5334102d90 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.092+0000 7f5339c73700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53341023d0 0x7f533419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f5339c73700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53341023d0 0x7f533419f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44156/0 (socket says 172.21.15.67:44156) 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f5339c73700 1 -- 172.21.15.67:0/3971089513 learned_addr learned my addr 172.21.15.67:0/3971089513 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f5339472700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5334102d90 0x7f533419fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:51.096 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f533a474700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5334108450 0x7f53341999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f5339c73700 1 -- 172.21.15.67:0/3971089513 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5334102d90 msgr2=0x7f533419fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f5339c73700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5334102d90 0x7f533419fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f5339c73700 1 -- 172.21.15.67:0/3971089513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5334108450 msgr2=0x7f53341999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f5339c73700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5334108450 0x7f53341999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f5339c73700 1 -- 172.21.15.67:0/3971089513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533419a110 con 0x7f53341023d0 2024-05-03T17:23:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f5339c73700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53341023d0 0x7f533419f7f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5330000c00 tx=0x7f5330003a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f532affd700 1 -- 172.21.15.67:0/3971089513 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5330027070 con 0x7f53341023d0 2024-05-03T17:23:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f532affd700 1 -- 172.21.15.67:0/3971089513 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5330003680 con 0x7f53341023d0 2024-05-03T17:23:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f533bed7700 1 -- 172.21.15.67:0/3971089513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f533419a3a0 con 0x7f53341023d0 2024-05-03T17:23:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f532affd700 1 -- 172.21.15.67:0/3971089513 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5330022c70 con 0x7f53341023d0 2024-05-03T17:23:51.097 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.093+0000 7f533bed7700 1 -- 172.21.15.67:0/3971089513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53341a66c0 con 0x7f53341023d0 2024-05-03T17:23:51.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.094+0000 7f533bed7700 1 -- 172.21.15.67:0/3971089513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5334103ac0 con 0x7f53341023d0 2024-05-03T17:23:51.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.096+0000 7f532affd700 1 -- 172.21.15.67:0/3971089513 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f533002a070 con 0x7f53341023d0 2024-05-03T17:23:51.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.097+0000 7f532affd700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f53200b2990 0x7f53200b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:51.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.097+0000 7f5339472700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f53200b2990 0x7f53200b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:51.098 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.097+0000 7f532affd700 1 -- 172.21.15.67:0/3971089513 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f53300e30a0 con 0x7f53341023d0 2024-05-03T17:23:51.099 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.097+0000 7f5339472700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f53200b2990 0x7f53200b4e50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5334078980 tx=0x7f532400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:51.103 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.101+0000 7f532affd700 1 -- 172.21.15.67:0/3971089513 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f5330013070 con 0x7f53341023d0 2024-05-03T17:23:51.367 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.366+0000 7f533bed7700 1 -- 172.21.15.67:0/3971089513 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f53341a6890 con 0x7f53341023d0 2024-05-03T17:23:51.368 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.366+0000 7f532affd700 1 -- 172.21.15.67:0/3971089513 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) ==== 53+0+73 (secure 0 0 0) 0x7f53300aace0 con 0x7f53341023d0 2024-05-03T17:23:51.368 INFO:teuthology.orchestra.run.smithi067.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-03T17:23:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.368+0000 7f533bed7700 1 -- 172.21.15.67:0/3971089513 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f53200b2990 msgr2=0x7f53200b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.368+0000 7f533bed7700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f53200b2990 0x7f53200b4e50 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5334078980 tx=0x7f532400b410 comp rx=0 tx=0).stop 2024-05-03T17:23:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.368+0000 7f533bed7700 1 -- 172.21.15.67:0/3971089513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53341023d0 msgr2=0x7f533419f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.368+0000 7f533bed7700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53341023d0 0x7f533419f7f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5330000c00 tx=0x7f5330003a90 comp rx=0 tx=0).stop 2024-05-03T17:23:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.368+0000 7f533bed7700 1 -- 172.21.15.67:0/3971089513 shutdown_connections 2024-05-03T17:23:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.368+0000 7f533bed7700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f53200b2990 0x7f53200b4e50 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.368+0000 7f533bed7700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f5334108450 0x7f53341999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.368+0000 7f533bed7700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f53341023d0 0x7f533419f7f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:51.370 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.368+0000 7f533bed7700 1 --2- 172.21.15.67:0/3971089513 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f5334102d90 0x7f533419fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.368+0000 7f533bed7700 1 -- 172.21.15.67:0/3971089513 >> 172.21.15.67:0/3971089513 conn(0x7f5334078410 msgr2=0x7f5334101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.369+0000 7f533bed7700 1 -- 172.21.15.67:0/3971089513 shutdown_connections 2024-05-03T17:23:51.371 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:51.369+0000 7f533bed7700 1 -- 172.21.15.67:0/3971089513 wait complete. 2024-05-03T17:23:52.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:51 smithi082 ceph-mon[33535]: pgmap v393: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:52.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:51 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3971089513' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-03T17:23:52.052 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:51 smithi067 ceph-mon[29731]: pgmap v393: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:52.052 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:51 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3971089513' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-03T17:23:52.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:51 smithi070 ceph-mon[33250]: pgmap v393: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:23:52.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:51 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3971089513' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-03T17:23:52.081 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-03T17:23:52.081 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-03T17:23:52.081 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi067.front.sepia.ceph.com 2024-05-03T17:23:52.081 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-03T17:23:52.081 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-03T17:23:52.081 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-03T17:23:52.081 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-03T17:23:52.081 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-03T17:23:52.081 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-03T17:23:52.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:23:52.082 DEBUG:teuthology.orchestra.run.smithi067:> ip addr 2024-05-03T17:23:52.099 INFO:teuthology.orchestra.run.smithi067.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-03T17:23:52.099 INFO:teuthology.orchestra.run.smithi067.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-03T17:23:52.099 INFO:teuthology.orchestra.run.smithi067.stdout: inet 127.0.0.1/8 scope host lo 2024-05-03T17:23:52.099 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-03T17:23:52.099 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 ::1/128 scope host 2024-05-03T17:23:52.099 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-03T17:23:52.099 INFO:teuthology.orchestra.run.smithi067.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-03T17:23:52.099 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7e brd ff:ff:ff:ff:ff:ff 2024-05-03T17:23:52.099 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f0 2024-05-03T17:23:52.099 INFO:teuthology.orchestra.run.smithi067.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-03T17:23:52.100 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:a8:b4:7f brd ff:ff:ff:ff:ff:ff 2024-05-03T17:23:52.100 INFO:teuthology.orchestra.run.smithi067.stdout: altname enp7s0f1 2024-05-03T17:23:52.100 INFO:teuthology.orchestra.run.smithi067.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-03T17:23:52.100 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0a brd ff:ff:ff:ff:ff:ff 2024-05-03T17:23:52.100 INFO:teuthology.orchestra.run.smithi067.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-03T17:23:52.100 INFO:teuthology.orchestra.run.smithi067.stdout: link/ether 0c:c4:7a:b6:fc:0b brd ff:ff:ff:ff:ff:ff 2024-05-03T17:23:52.100 INFO:teuthology.orchestra.run.smithi067.stdout: inet 172.21.15.67/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-03T17:23:52.100 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft 41999sec preferred_lft 41999sec 2024-05-03T17:23:52.100 INFO:teuthology.orchestra.run.smithi067.stdout: inet6 fe80::2e0f:da1f:ecf7:40f6/64 scope link noprefixroute 2024-05-03T17:23:52.100 INFO:teuthology.orchestra.run.smithi067.stdout: valid_lft forever preferred_lft forever 2024-05-03T17:23:52.100 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-03T17:23:52.101 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-03T17:23:52.101 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-03T17:23:52.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add name ceph-brx type bridge 2024-05-03T17:23:52.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr flush dev ceph-brx 2024-05-03T17:23:52.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set ceph-brx up 2024-05-03T17:23:52.101 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-03T17:23:52.101 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-03T17:23:52.189 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-03T17:23:52.278 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T17:23:52.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:23:52.285 DEBUG:teuthology.orchestra.run.smithi067:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-03T17:23:52.310 INFO:teuthology.orchestra.run.smithi067.stdout:1 2024-05-03T17:23:52.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:23:52.312 DEBUG:teuthology.orchestra.run.smithi067:> route 2024-05-03T17:23:52.376 INFO:teuthology.orchestra.run.smithi067.stdout:Kernel IP routing table 2024-05-03T17:23:52.376 INFO:teuthology.orchestra.run.smithi067.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-03T17:23:52.376 INFO:teuthology.orchestra.run.smithi067.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-03T17:23:52.377 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-03T17:23:52.377 INFO:teuthology.orchestra.run.smithi067.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-03T17:23:52.377 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-03T17:23:52.377 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-03T17:23:52.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-03T17:23:52.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-03T17:23:52.377 DEBUG:teuthology.orchestra.run.smithi067:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-03T17:23:52.377 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-03T17:23:52.466 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-03T17:23:52.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T17:23:52.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:23:52.665 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-03T17:23:52.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:23:52.681 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list-id 2024-05-03T17:23:52.736 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-03T17:23:52.736 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-03T17:23:52.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-03T17:23:52.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-03T17:23:52.736 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-03T17:23:52.826 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-03T17:23:52.849 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T17:23:52.855 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-03T17:23:52.855 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-03T17:23:52.856 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-03T17:23:52.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-03T17:23:52.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-03T17:23:52.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-03T17:23:52.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-03T17:23:52.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-03T17:23:52.856 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-03T17:23:52.905 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-03T17:23:52.977 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T17:23:52.983 DEBUG:teuthology.orchestra.run.smithi067:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-03T17:23:52.983 DEBUG:teuthology.orchestra.run.smithi067:> set -e 2024-05-03T17:23:52.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set brx.0 up 2024-05-03T17:23:52.983 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link set dev brx.0 master ceph-brx 2024-05-03T17:23:52.984 DEBUG:teuthology.orchestra.run.smithi067:> ') 2024-05-03T17:23:53.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-03T17:23:53.061 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T17:23:53.068 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-03T17:23:53.068 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-03T17:23:53.083 INFO:teuthology.orchestra.run.smithi067.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-03T17:23:53.083 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-03T17:23:53.083 DEBUG:teuthology.orchestra.run.smithi067:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-03T17:23:53.138 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-05-03T17:23:53.138 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-05-03T17:23:53.139 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/32cff0e8-2085-4382-86b8-36aa14853e65 2024-05-03T17:23:53.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:23:53.139 DEBUG:teuthology.orchestra.run.smithi067:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/32cff0e8-2085-4382-86b8-36aa14853e65 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-05-03T17:23:53.245 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-05-03T17:23:53.245 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-05-03T17:23:53.245 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-05-03T17:23:53.362 INFO:teuthology.orchestra.run.smithi067.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-05-03T17:23:53.362 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-05-03T17:23:53.362 INFO:teuthology.orchestra.run.smithi067.stdout:invalid new device string format 2024-05-03T17:23:53.362 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: resolved to: "172.21.15.67:3300,172.21.15.70:3300,172.21.15.82:3300" 2024-05-03T17:23:53.362 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.67:3300,172.21.15.70:3300,172.21.15.82:3300:/volumes/qa/sv_1/32cff0e8-2085-4382-86b8-36aa14853e65 2024-05-03T17:23:53.363 INFO:teuthology.orchestra.run.smithi067.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=20c8e22e-0970-11ef-bc97-c7b262605968" will pass to kernel 2024-05-03T17:23:53.363 INFO:teuthology.orchestra.run.smithi067.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-03T17:23:53.363 INFO:teuthology.orchestra.run.smithi067.stdout: You just mounted a file system that supports labels which does not 2024-05-03T17:23:53.363 INFO:teuthology.orchestra.run.smithi067.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-03T17:23:53.363 INFO:teuthology.orchestra.run.smithi067.stdout: applications will generate AVC messages and not be allowed access to 2024-05-03T17:23:53.364 INFO:teuthology.orchestra.run.smithi067.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-03T17:23:53.365 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-03T17:23:53.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:23:53.366 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-03T17:23:53.391 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-03T17:23:53.484 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-03T17:23:53.486 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T17:23:53.492 INFO:teuthology.orchestra.run.smithi067.stdout:client24830 2024-05-03T17:23:53.492 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:23:53.492 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-03T17:23:53.509 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-03T17:23:53.599 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-03T17:23:53.601 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T17:23:53.608 INFO:teuthology.orchestra.run.smithi067.stdout:client24830 2024-05-03T17:23:53.608 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-03T17:23:53.657 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-03T17:23:53.659 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T17:23:53.665 INFO:teuthology.orchestra.run.smithi067.stdout:client24830 2024-05-03T17:23:53.665 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/20c8e22e-0970-11ef-bc97-c7b262605968.client24830/status') 2024-05-03T17:23:53.715 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-03T17:23:53.725 INFO:teuthology.orchestra.run.smithi067.stdout:instance: client.24830 (3)172.21.15.67:0/3897340080 2024-05-03T17:23:53.726 INFO:teuthology.orchestra.run.smithi067.stdout:blocklisted: false 2024-05-03T17:23:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records in 2024-05-03T17:23:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:0+1 records out 2024-05-03T17:23:53.726 INFO:teuthology.orchestra.run.smithi067.stderr:71 bytes copied, 2.4239e-05 s, 2.9 MB/s 2024-05-03T17:23:53.727 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-03T17:23:53.733 INFO:teuthology.run_tasks:Running task exec... 2024-05-03T17:23:53.744 INFO:teuthology.task.exec:Executing custom commands... 2024-05-03T17:23:53.744 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi067.front.sepia.ceph.com 2024-05-03T17:23:53.744 DEBUG:teuthology.orchestra.run.smithi067:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-05-03T17:23:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.848+0000 7ff2d83c8640 1 -- 172.21.15.67:0/974714626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2d0102b00 msgr2=0x7ff2d0102f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.848+0000 7ff2d83c8640 1 --2- 172.21.15.67:0/974714626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2d0102b00 0x7ff2d0102f80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc009900 tx=0x7ff2cc02a410 comp rx=0 tx=0).stop 2024-05-03T17:23:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.848+0000 7ff2d83c8640 1 -- 172.21.15.67:0/974714626 shutdown_connections 2024-05-03T17:23:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.848+0000 7ff2d83c8640 1 --2- 172.21.15.67:0/974714626 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff2d01034c0 0x7ff2d0109d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.848+0000 7ff2d83c8640 1 --2- 172.21.15.67:0/974714626 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2d0102b00 0x7ff2d0102f80 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.848+0000 7ff2d83c8640 1 --2- 172.21.15.67:0/974714626 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2d0101910 0x7ff2d0101d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.848+0000 7ff2d83c8640 1 -- 172.21.15.67:0/974714626 >> 172.21.15.67:0/974714626 conn(0x7ff2d00fce80 msgr2=0x7ff2d00ff2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:53.850 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.848+0000 7ff2d83c8640 1 -- 172.21.15.67:0/974714626 shutdown_connections 2024-05-03T17:23:53.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.848+0000 7ff2d83c8640 1 -- 172.21.15.67:0/974714626 wait complete. 2024-05-03T17:23:53.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d83c8640 1 Processor -- start 2024-05-03T17:23:53.851 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d83c8640 1 -- start start 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d83c8640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2d0101910 0x7ff2d0198cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d83c8640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2d0102b00 0x7ff2d01991f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d83c8640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff2d01034c0 0x7ff2d01a01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d83c8640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d010d490 con 0x7ff2d0102b00 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d83c8640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d010d310 con 0x7ff2d0101910 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d83c8640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7ff2d010d610 con 0x7ff2d01034c0 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d5963640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2d0102b00 0x7ff2d01991f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d5963640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2d0102b00 0x7ff2d01991f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44178/0 (socket says 172.21.15.67:44178) 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d5963640 1 -- 172.21.15.67:0/963075122 learned_addr learned my addr 172.21.15.67:0/963075122 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d6965640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff2d01034c0 0x7ff2d01a01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d6164640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2d0101910 0x7ff2d0198cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d5963640 1 -- 172.21.15.67:0/963075122 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff2d01034c0 msgr2=0x7ff2d01a01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d5963640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff2d01034c0 0x7ff2d01a01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:53.853 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d5963640 1 -- 172.21.15.67:0/963075122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2d0101910 msgr2=0x7ff2d0198cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d5963640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2d0101910 0x7ff2d0198cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.849+0000 7ff2d5963640 1 -- 172.21.15.67:0/963075122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2d01a0870 con 0x7ff2d0102b00 2024-05-03T17:23:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.850+0000 7ff2d5963640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2d0102b00 0x7ff2d01991f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc005ab0 tx=0x7ff2cc003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.850+0000 7ff2c77fe640 1 -- 172.21.15.67:0/963075122 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc03f070 con 0x7ff2d0102b00 2024-05-03T17:23:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.850+0000 7ff2d83c8640 1 -- 172.21.15.67:0/963075122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2d01a0b00 con 0x7ff2d0102b00 2024-05-03T17:23:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.850+0000 7ff2c77fe640 1 -- 172.21.15.67:0/963075122 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff2cc031c50 con 0x7ff2d0102b00 2024-05-03T17:23:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.850+0000 7ff2c77fe640 1 -- 172.21.15.67:0/963075122 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2cc035030 con 0x7ff2d0102b00 2024-05-03T17:23:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.850+0000 7ff2d83c8640 1 -- 172.21.15.67:0/963075122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2d01a1040 con 0x7ff2d0102b00 2024-05-03T17:23:53.854 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.851+0000 7ff2d83c8640 1 -- 172.21.15.67:0/963075122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2d01a1270 con 0x7ff2d0102b00 2024-05-03T17:23:53.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.852+0000 7ff2c77fe640 1 -- 172.21.15.67:0/963075122 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7ff2cc042070 con 0x7ff2d0102b00 2024-05-03T17:23:53.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.853+0000 7ff2c77fe640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff2bc0b2870 0x7ff2bc0b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:53.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.853+0000 7ff2d6164640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff2bc0b2870 0x7ff2bc0b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:53.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.853+0000 7ff2c77fe640 1 -- 172.21.15.67:0/963075122 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7ff2cc1033c0 con 0x7ff2d0102b00 2024-05-03T17:23:53.855 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.854+0000 7ff2d6164640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff2bc0b2870 0x7ff2bc0b4d30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff2d019a130 tx=0x7ff2c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:53.858 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:53.857+0000 7ff2c77fe640 1 -- 172.21.15.67:0/963075122 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7ff2cc0caf10 con 0x7ff2d0102b00 2024-05-03T17:23:53.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:53 smithi067 ceph-mon[29731]: pgmap v394: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:54.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:53 smithi070 ceph-mon[33250]: pgmap v394: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:54.093 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.091+0000 7ff2d83c8640 1 -- 172.21.15.67:0/963075122 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7ff2d01a1440 con 0x7ff2d0102b00 2024-05-03T17:23:54.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:53 smithi082 ceph-mon[33535]: pgmap v394: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:23:54.772 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.770+0000 7ff2c77fe640 1 -- 172.21.15.67:0/963075122 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v26) ==== 118+0+0 (secure 0 0 0) 0x7ff2cc0ca810 con 0x7ff2d0102b00 2024-05-03T17:23:54.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.774+0000 7ff2d83c8640 1 -- 172.21.15.67:0/963075122 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff2bc0b2870 msgr2=0x7ff2bc0b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:54.775 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.774+0000 7ff2d83c8640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff2bc0b2870 0x7ff2bc0b4d30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff2d019a130 tx=0x7ff2c0005f90 comp rx=0 tx=0).stop 2024-05-03T17:23:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.774+0000 7ff2d83c8640 1 -- 172.21.15.67:0/963075122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2d0102b00 msgr2=0x7ff2d01991f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.774+0000 7ff2d83c8640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2d0102b00 0x7ff2d01991f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc005ab0 tx=0x7ff2cc003b90 comp rx=0 tx=0).stop 2024-05-03T17:23:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.774+0000 7ff2d83c8640 1 -- 172.21.15.67:0/963075122 shutdown_connections 2024-05-03T17:23:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.774+0000 7ff2d83c8640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7ff2d01034c0 0x7ff2d01a01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.774+0000 7ff2d83c8640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff2d0102b00 0x7ff2d01991f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.774+0000 7ff2d83c8640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7ff2bc0b2870 0x7ff2bc0b4d30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.774+0000 7ff2d83c8640 1 --2- 172.21.15.67:0/963075122 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7ff2d0101910 0x7ff2d0198cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.774+0000 7ff2d83c8640 1 -- 172.21.15.67:0/963075122 >> 172.21.15.67:0/963075122 conn(0x7ff2d00fce80 msgr2=0x7ff2d00fe990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.774+0000 7ff2d83c8640 1 -- 172.21.15.67:0/963075122 shutdown_connections 2024-05-03T17:23:54.776 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.774+0000 7ff2d83c8640 1 -- 172.21.15.67:0/963075122 wait complete. 2024-05-03T17:23:54.787 DEBUG:teuthology.orchestra.run.smithi067:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-05-03T17:23:54.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.893+0000 7f57536da640 1 -- 172.21.15.67:0/3800884456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c1034c0 msgr2=0x7f574c1058b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:54.897 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.893+0000 7f57536da640 1 --2- 172.21.15.67:0/3800884456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c1034c0 0x7f574c1058b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f5748009900 tx=0x7f574802a410 comp rx=0 tx=0).stop 2024-05-03T17:23:54.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 -- 172.21.15.67:0/3800884456 shutdown_connections 2024-05-03T17:23:54.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 --2- 172.21.15.67:0/3800884456 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f574c108720 0x7f574c10ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:54.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 --2- 172.21.15.67:0/3800884456 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f574c105df0 0x7f574c1081e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:54.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 --2- 172.21.15.67:0/3800884456 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c1034c0 0x7f574c1058b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:54.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 -- 172.21.15.67:0/3800884456 >> 172.21.15.67:0/3800884456 conn(0x7f574c0fced0 msgr2=0x7f574c0ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:23:54.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 -- 172.21.15.67:0/3800884456 shutdown_connections 2024-05-03T17:23:54.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 -- 172.21.15.67:0/3800884456 wait complete. 2024-05-03T17:23:54.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 Processor -- start 2024-05-03T17:23:54.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 -- start start 2024-05-03T17:23:54.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c1034c0 0x7f574c102e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:54.898 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f574c105df0 0x7f574c101470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f574c108720 0x7f574c1019b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f574c10e5f0 con 0x7f574c1034c0 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f574c10e470 con 0x7f574c105df0 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.894+0000 7f57536da640 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f574c10e770 con 0x7f574c108720 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f5751476640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c1034c0 0x7f574c102e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f5751476640 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c1034c0 0x7f574c102e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:44204/0 (socket says 172.21.15.67:44204) 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f5751476640 1 -- 172.21.15.67:0/1530589646 learned_addr learned my addr 172.21.15.67:0/1530589646 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f5751c77640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f574c108720 0x7f574c1019b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f5750c75640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f574c105df0 0x7f574c101470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f5751476640 1 -- 172.21.15.67:0/1530589646 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f574c108720 msgr2=0x7f574c1019b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f5751476640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f574c108720 0x7f574c1019b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f5751476640 1 -- 172.21.15.67:0/1530589646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f574c105df0 msgr2=0x7f574c101470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f5751476640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f574c105df0 0x7f574c101470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f5751476640 1 -- 172.21.15.67:0/1530589646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574c102240 con 0x7f574c1034c0 2024-05-03T17:23:54.899 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f5750c75640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f574c105df0 0x7f574c101470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:23:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f5751476640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c1034c0 0x7f574c102e30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f5748005940 tx=0x7f5748004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f574803f030 con 0x7f574c1034c0 2024-05-03T17:23:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f57536da640 1 -- 172.21.15.67:0/1530589646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f574c1a2df0 con 0x7f574c1034c0 2024-05-03T17:23:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5748003cb0 con 0x7f574c1034c0 2024-05-03T17:23:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5748035070 con 0x7f574c1034c0 2024-05-03T17:23:54.900 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.895+0000 7f57536da640 1 -- 172.21.15.67:0/1530589646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f574c1a32d0 con 0x7f574c1034c0 2024-05-03T17:23:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.896+0000 7f57536da640 1 -- 172.21.15.67:0/1530589646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f574c0fe9c0 con 0x7f574c1034c0 2024-05-03T17:23:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.898+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f5748042030 con 0x7f574c1034c0 2024-05-03T17:23:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.898+0000 7f57427fc640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f57380b2870 0x7f57380b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:23:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.898+0000 7f5750c75640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f57380b2870 0x7f57380b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:23:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.899+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f5748102ed0 con 0x7f574c1034c0 2024-05-03T17:23:54.901 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.899+0000 7f5750c75640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f57380b2870 0x7f57380b4d30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f573c006fd0 tx=0x7f573c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:23:54.903 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:54.902+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f57480caa80 con 0x7f574c1034c0 2024-05-03T17:23:55.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:54 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/963075122' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-03T17:23:55.124 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:23:55.122+0000 7f57536da640 1 -- 172.21.15.67:0/1530589646 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f574c062e90 con 0x7f57380b2870 2024-05-03T17:23:55.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:54 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/963075122' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-03T17:23:55.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:54 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/963075122' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-05-03T17:23:56.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:55 smithi070 ceph-mon[33250]: pgmap v395: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-03T17:23:56.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:55 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/963075122' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-03T17:23:56.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:55 smithi070 ceph-mon[33250]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T17:23:56.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:55 smithi067 ceph-mon[29731]: pgmap v395: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-03T17:23:56.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:55 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/963075122' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-03T17:23:56.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:55 smithi067 ceph-mon[29731]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T17:23:56.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:55 smithi082 ceph-mon[33535]: pgmap v395: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2024-05-03T17:23:56.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:55 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/963075122' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-05-03T17:23:56.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:55 smithi082 ceph-mon[33535]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-05-03T17:23:57.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:56 smithi070 ceph-mon[33250]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:57.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:56 smithi067 ceph-mon[29731]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:57.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:56 smithi082 ceph-mon[33535]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:23:58.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:57 smithi070 ceph-mon[33250]: pgmap v396: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:23:58.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:57 smithi067 ceph-mon[29731]: pgmap v396: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:23:58.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:57 smithi082 ceph-mon[33535]: pgmap v396: 97 pgs: 97 active+clean; 631 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:24:00.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:23:59 smithi070 ceph-mon[33250]: pgmap v397: 97 pgs: 97 active+clean; 632 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-05-03T17:24:00.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:23:59 smithi067 ceph-mon[29731]: pgmap v397: 97 pgs: 97 active+clean; 632 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-05-03T17:24:00.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:23:59 smithi082 ceph-mon[33535]: pgmap v397: 97 pgs: 97 active+clean; 632 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2024-05-03T17:24:02.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:01 smithi070 ceph-mon[33250]: pgmap v398: 97 pgs: 97 active+clean; 632 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-03T17:24:02.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:01 smithi067 ceph-mon[29731]: pgmap v398: 97 pgs: 97 active+clean; 632 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-03T17:24:02.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:01 smithi082 ceph-mon[33535]: pgmap v398: 97 pgs: 97 active+clean; 632 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-03T17:24:04.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:03 smithi070 ceph-mon[33250]: pgmap v399: 97 pgs: 97 active+clean; 632 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-03T17:24:04.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:03 smithi067 ceph-mon[29731]: pgmap v399: 97 pgs: 97 active+clean; 632 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-03T17:24:04.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:03 smithi082 ceph-mon[33535]: pgmap v399: 97 pgs: 97 active+clean; 632 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2024-05-03T17:24:06.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:05 smithi070 ceph-mon[33250]: pgmap v400: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 853 B/s wr, 7 op/s 2024-05-03T17:24:06.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:05 smithi067 ceph-mon[29731]: pgmap v400: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 853 B/s wr, 7 op/s 2024-05-03T17:24:06.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:05 smithi082 ceph-mon[33535]: pgmap v400: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 853 B/s wr, 7 op/s 2024-05-03T17:24:08.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:07 smithi070 ceph-mon[33250]: pgmap v401: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 5 op/s 2024-05-03T17:24:08.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:07 smithi067 ceph-mon[29731]: pgmap v401: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 5 op/s 2024-05-03T17:24:08.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:07 smithi082 ceph-mon[33535]: pgmap v401: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 5 op/s 2024-05-03T17:24:10.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:09 smithi070 ceph-mon[33250]: pgmap v402: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-05-03T17:24:10.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:09 smithi067 ceph-mon[29731]: pgmap v402: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-05-03T17:24:10.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:09 smithi082 ceph-mon[33535]: pgmap v402: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 767 B/s wr, 7 op/s 2024-05-03T17:24:12.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:11 smithi067 ceph-mon[29731]: pgmap v403: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-03T17:24:12.265 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:11 smithi082 ceph-mon[33535]: pgmap v403: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-03T17:24:12.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:11 smithi070 ceph-mon[33250]: pgmap v403: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-03T17:24:14.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:14 smithi070 ceph-mon[33250]: pgmap v404: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-03T17:24:14.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:14 smithi067 ceph-mon[29731]: pgmap v404: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-03T17:24:14.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:14 smithi082 ceph-mon[33535]: pgmap v404: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 5 op/s 2024-05-03T17:24:16.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:16 smithi070 ceph-mon[33250]: pgmap v405: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-03T17:24:16.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:16 smithi067 ceph-mon[29731]: pgmap v405: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-03T17:24:16.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:16 smithi082 ceph-mon[33535]: pgmap v405: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-03T17:24:18.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:18 smithi070 ceph-mon[33250]: pgmap v406: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:18.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:18 smithi067 ceph-mon[29731]: pgmap v406: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:18.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:18 smithi082 ceph-mon[33535]: pgmap v406: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:20.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:20 smithi070 ceph-mon[33250]: pgmap v407: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:24:20.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:20 smithi067 ceph-mon[29731]: pgmap v407: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:24:20.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:20 smithi082 ceph-mon[33535]: pgmap v407: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:24:22.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:22 smithi070 ceph-mon[33250]: pgmap v408: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:24:22.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:22 smithi067 ceph-mon[29731]: pgmap v408: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:24:22.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:22 smithi082 ceph-mon[33535]: pgmap v408: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:24:24.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:24 smithi070 ceph-mon[33250]: pgmap v409: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:24:24.317 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:24 smithi082 ceph-mon[33535]: pgmap v409: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:24:24.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:24 smithi067 ceph-mon[29731]: pgmap v409: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:24:26.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:26 smithi070 ceph-mon[33250]: pgmap v410: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:24:26.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:26 smithi067 ceph-mon[29731]: pgmap v410: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:24:26.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:26 smithi082 ceph-mon[33535]: pgmap v410: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:24:28.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:28 smithi070 ceph-mon[33250]: pgmap v411: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:28.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:28 smithi067 ceph-mon[29731]: pgmap v411: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:28.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:28 smithi082 ceph-mon[33535]: pgmap v411: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:30.308 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:30 smithi070 ceph-mon[33250]: pgmap v412: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:24:30.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:30 smithi067 ceph-mon[29731]: pgmap v412: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:24:30.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:30 smithi082 ceph-mon[33535]: pgmap v412: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:24:32.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:32 smithi067 ceph-mon[29731]: pgmap v413: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:24:32.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:32 smithi082 ceph-mon[33535]: pgmap v413: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:24:32.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:32 smithi070 ceph-mon[33250]: pgmap v413: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:24:34.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:34 smithi067 ceph-mon[29731]: pgmap v414: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:24:34.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:34 smithi082 ceph-mon[33535]: pgmap v414: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:24:34.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:34 smithi070 ceph-mon[33250]: pgmap v414: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:24:36.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:36 smithi067 ceph-mon[29731]: pgmap v415: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:24:36.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:36 smithi082 ceph-mon[33535]: pgmap v415: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:24:36.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:36 smithi070 ceph-mon[33250]: pgmap v415: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:24:38.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:38 smithi067 ceph-mon[29731]: pgmap v416: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:38.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:38 smithi082 ceph-mon[33535]: pgmap v416: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:38.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:38 smithi070 ceph-mon[33250]: pgmap v416: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:40.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:40 smithi067 ceph-mon[29731]: pgmap v417: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:24:40.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:40 smithi082 ceph-mon[33535]: pgmap v417: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:24:40.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:40 smithi070 ceph-mon[33250]: pgmap v417: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:24:42.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:42 smithi067 ceph-mon[29731]: pgmap v418: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:24:42.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:42 smithi082 ceph-mon[33535]: pgmap v418: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:24:42.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:42 smithi070 ceph-mon[33250]: pgmap v418: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:24:44.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:44 smithi067 ceph-mon[29731]: pgmap v419: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:24:44.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:44 smithi082 ceph-mon[33535]: pgmap v419: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:24:44.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:44 smithi070 ceph-mon[33250]: pgmap v419: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:24:46.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:46 smithi067 ceph-mon[29731]: pgmap v420: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:24:46.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:24:46.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:24:46.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:24:46.454 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:24:46.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:46 smithi082 ceph-mon[33535]: pgmap v420: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:24:46.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:24:46.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:24:46.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:24:46.515 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:24:46.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:46 smithi070 ceph-mon[33250]: pgmap v420: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:24:46.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:24:46.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:24:46.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:24:46.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:24:48.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:48 smithi067 ceph-mon[29731]: pgmap v421: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:48.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:48 smithi082 ceph-mon[33535]: pgmap v421: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:48.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:48 smithi070 ceph-mon[33250]: pgmap v421: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:50.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:50 smithi067 ceph-mon[29731]: pgmap v422: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:24:50.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:50 smithi082 ceph-mon[33535]: pgmap v422: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:24:50.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:50 smithi070 ceph-mon[33250]: pgmap v422: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:24:52.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:52 smithi067 ceph-mon[29731]: pgmap v423: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:24:52.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:52 smithi082 ceph-mon[33535]: pgmap v423: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:24:52.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:52 smithi070 ceph-mon[33250]: pgmap v423: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:24:54.360 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:54 smithi070 ceph-mon[33250]: pgmap v424: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:24:54.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:54 smithi067 ceph-mon[29731]: pgmap v424: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:24:54.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:54 smithi082 ceph-mon[33535]: pgmap v424: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:24:56.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:56 smithi067 ceph-mon[29731]: pgmap v425: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:24:56.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:56 smithi082 ceph-mon[33535]: pgmap v425: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:24:56.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:56 smithi070 ceph-mon[33250]: pgmap v425: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:24:58.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:24:58 smithi067 ceph-mon[29731]: pgmap v426: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:58.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:24:58 smithi082 ceph-mon[33535]: pgmap v426: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:24:58.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:24:58 smithi070 ceph-mon[33250]: pgmap v426: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:00.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:00 smithi067 ceph-mon[29731]: pgmap v427: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:00.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:00 smithi082 ceph-mon[33535]: pgmap v427: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:00.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:00 smithi070 ceph-mon[33250]: pgmap v427: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:02.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:02 smithi067 ceph-mon[29731]: pgmap v428: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:02.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:02 smithi082 ceph-mon[33535]: pgmap v428: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:02.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:02 smithi070 ceph-mon[33250]: pgmap v428: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:04.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:04 smithi067 ceph-mon[29731]: pgmap v429: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:04.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:04 smithi082 ceph-mon[33535]: pgmap v429: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:04.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:04 smithi070 ceph-mon[33250]: pgmap v429: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:06.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:06 smithi067 ceph-mon[29731]: pgmap v430: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:06.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:06 smithi082 ceph-mon[33535]: pgmap v430: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:06.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:06 smithi070 ceph-mon[33250]: pgmap v430: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:08.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:08 smithi067 ceph-mon[29731]: pgmap v431: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:08.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:08 smithi082 ceph-mon[33535]: pgmap v431: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:08.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:08 smithi070 ceph-mon[33250]: pgmap v431: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:10.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:10 smithi067 ceph-mon[29731]: pgmap v432: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:10.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:10 smithi082 ceph-mon[33535]: pgmap v432: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:10.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:10 smithi070 ceph-mon[33250]: pgmap v432: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:12.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:12 smithi067 ceph-mon[29731]: pgmap v433: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:12.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:12 smithi082 ceph-mon[33535]: pgmap v433: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:12.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:12 smithi070 ceph-mon[33250]: pgmap v433: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:14.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:14 smithi067 ceph-mon[29731]: pgmap v434: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:14.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:14 smithi082 ceph-mon[33535]: pgmap v434: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:14.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:14 smithi070 ceph-mon[33250]: pgmap v434: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:16.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:16 smithi067 ceph-mon[29731]: pgmap v435: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:16.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:16 smithi082 ceph-mon[33535]: pgmap v435: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:16.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:16 smithi070 ceph-mon[33250]: pgmap v435: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:18.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:18 smithi067 ceph-mon[29731]: pgmap v436: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:18.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:18 smithi082 ceph-mon[33535]: pgmap v436: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:18.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:18 smithi070 ceph-mon[33250]: pgmap v436: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:20.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:20 smithi067 ceph-mon[29731]: pgmap v437: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:20.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:20 smithi082 ceph-mon[33535]: pgmap v437: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:20.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:20 smithi070 ceph-mon[33250]: pgmap v437: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:21.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:21 smithi067 ceph-mon[29731]: pgmap v438: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:21.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:21 smithi082 ceph-mon[33535]: pgmap v438: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:21.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:21 smithi070 ceph-mon[33250]: pgmap v438: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:23.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:23 smithi070 ceph-mon[33250]: pgmap v439: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:23.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:23 smithi067 ceph-mon[29731]: pgmap v439: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:24.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:23 smithi082 ceph-mon[33535]: pgmap v439: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:25.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:25 smithi070 ceph-mon[33250]: pgmap v440: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:25.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:25 smithi067 ceph-mon[29731]: pgmap v440: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:26.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:25 smithi082 ceph-mon[33535]: pgmap v440: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:27.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:27 smithi070 ceph-mon[33250]: pgmap v441: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:27.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:27 smithi067 ceph-mon[29731]: pgmap v441: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:28.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:27 smithi082 ceph-mon[33535]: pgmap v441: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:29.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:29 smithi070 ceph-mon[33250]: pgmap v442: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:29.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:29 smithi067 ceph-mon[29731]: pgmap v442: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:30.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:29 smithi082 ceph-mon[33535]: pgmap v442: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:31.808 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:31 smithi070 ceph-mon[33250]: pgmap v443: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:31 smithi067 ceph-mon[29731]: pgmap v443: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:32.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:31 smithi082 ceph-mon[33535]: pgmap v443: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:33.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:33 smithi067 ceph-mon[29731]: pgmap v444: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:34.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:33 smithi082 ceph-mon[33535]: pgmap v444: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:34.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:33 smithi070 ceph-mon[33250]: pgmap v444: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:35.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:35 smithi067 ceph-mon[29731]: pgmap v445: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:36.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:35 smithi082 ceph-mon[33535]: pgmap v445: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:36.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:35 smithi070 ceph-mon[33250]: pgmap v445: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:37.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:37 smithi067 ceph-mon[29731]: pgmap v446: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:38.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:37 smithi082 ceph-mon[33535]: pgmap v446: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:38.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:37 smithi070 ceph-mon[33250]: pgmap v446: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:39.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:39 smithi067 ceph-mon[29731]: pgmap v447: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:40.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:39 smithi082 ceph-mon[33535]: pgmap v447: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:40.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:39 smithi070 ceph-mon[33250]: pgmap v447: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:41.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:41 smithi067 ceph-mon[29731]: pgmap v448: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:42.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:41 smithi082 ceph-mon[33535]: pgmap v448: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:42.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:41 smithi070 ceph-mon[33250]: pgmap v448: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:43.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:43 smithi067 ceph-mon[29731]: pgmap v449: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:44.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:43 smithi082 ceph-mon[33535]: pgmap v449: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:44.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:43 smithi070 ceph-mon[33250]: pgmap v449: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:45.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:45 smithi067 ceph-mon[29731]: pgmap v450: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:46.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:45 smithi082 ceph-mon[33535]: pgmap v450: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:46.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:45 smithi070 ceph-mon[33250]: pgmap v450: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:46.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:46 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:25:47.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:46 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:25:47.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:46 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:25:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:47 smithi067 ceph-mon[29731]: pgmap v451: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:47.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:25:47.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:25:47.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:25:48.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:47 smithi082 ceph-mon[33535]: pgmap v451: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:48.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:47 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:25:48.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:47 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:25:48.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:47 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:25:48.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:47 smithi070 ceph-mon[33250]: pgmap v451: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:48.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:47 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:25:48.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:47 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:25:48.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:47 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:25:49.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:49 smithi067 ceph-mon[29731]: pgmap v452: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:50.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:49 smithi082 ceph-mon[33535]: pgmap v452: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:50.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:49 smithi070 ceph-mon[33250]: pgmap v452: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:25:51.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:51 smithi067 ceph-mon[29731]: pgmap v453: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:52.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:51 smithi082 ceph-mon[33535]: pgmap v453: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:52.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:51 smithi070 ceph-mon[33250]: pgmap v453: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:25:53.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:53 smithi067 ceph-mon[29731]: pgmap v454: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:54.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:53 smithi082 ceph-mon[33535]: pgmap v454: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:54.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:53 smithi070 ceph-mon[33250]: pgmap v454: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:25:55.860 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:55 smithi070 ceph-mon[33250]: pgmap v455: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:55.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:55 smithi067 ceph-mon[29731]: pgmap v455: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:56.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:55 smithi082 ceph-mon[33535]: pgmap v455: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:25:57.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:57 smithi067 ceph-mon[29731]: pgmap v456: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:58.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:57 smithi082 ceph-mon[33535]: pgmap v456: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:58.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:57 smithi070 ceph-mon[33250]: pgmap v456: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:25:59.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:25:59 smithi067 ceph-mon[29731]: pgmap v457: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:00.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:25:59 smithi082 ceph-mon[33535]: pgmap v457: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:00.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:25:59 smithi070 ceph-mon[33250]: pgmap v457: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:01.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:01 smithi067 ceph-mon[29731]: pgmap v458: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:26:02.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:01 smithi082 ceph-mon[33535]: pgmap v458: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:26:02.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:01 smithi070 ceph-mon[33250]: pgmap v458: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:26:03.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:03 smithi067 ceph-mon[29731]: pgmap v459: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:04.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:03 smithi082 ceph-mon[33535]: pgmap v459: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:04.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:03 smithi070 ceph-mon[33250]: pgmap v459: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:05.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:05 smithi067 ceph-mon[29731]: pgmap v460: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:26:06.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:05 smithi082 ceph-mon[33535]: pgmap v460: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:26:06.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:05 smithi070 ceph-mon[33250]: pgmap v460: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:26:07.954 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:07 smithi067 ceph-mon[29731]: pgmap v461: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:08.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:07 smithi082 ceph-mon[33535]: pgmap v461: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:08.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:07 smithi070 ceph-mon[33250]: pgmap v461: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:09.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:09 smithi067 ceph-mon[29731]: pgmap v462: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:10.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:09 smithi082 ceph-mon[33535]: pgmap v462: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:10.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:09 smithi070 ceph-mon[33250]: pgmap v462: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:11.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:11 smithi067 ceph-mon[29731]: pgmap v463: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:26:12.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:11 smithi082 ceph-mon[33535]: pgmap v463: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:26:12.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:11 smithi070 ceph-mon[33250]: pgmap v463: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:26:13.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:13 smithi067 ceph-mon[29731]: pgmap v464: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:14.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:13 smithi082 ceph-mon[33535]: pgmap v464: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:13 smithi070 ceph-mon[33250]: pgmap v464: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:15.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:15 smithi067 ceph-mon[29731]: pgmap v465: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:26:16.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:15 smithi082 ceph-mon[33535]: pgmap v465: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:26:16.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:15 smithi070 ceph-mon[33250]: pgmap v465: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:26:17.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:17 smithi067 ceph-mon[29731]: pgmap v466: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:18.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:17 smithi082 ceph-mon[33535]: pgmap v466: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:18.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:17 smithi070 ceph-mon[33250]: pgmap v466: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:19.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:19 smithi067 ceph-mon[29731]: pgmap v467: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:20.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:19 smithi082 ceph-mon[33535]: pgmap v467: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:20.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:19 smithi070 ceph-mon[33250]: pgmap v467: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:21.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:21 smithi067 ceph-mon[29731]: pgmap v468: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:26:22.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:21 smithi082 ceph-mon[33535]: pgmap v468: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:26:22.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:21 smithi070 ceph-mon[33250]: pgmap v468: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:26:23.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:23 smithi067 ceph-mon[29731]: pgmap v469: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:24.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:23 smithi082 ceph-mon[33535]: pgmap v469: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:24.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:23 smithi070 ceph-mon[33250]: pgmap v469: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:25.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:25 smithi067 ceph-mon[29731]: pgmap v470: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:26:26.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:25 smithi082 ceph-mon[33535]: pgmap v470: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:26:26.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:25 smithi070 ceph-mon[33250]: pgmap v470: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:26:27.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:27 smithi067 ceph-mon[29731]: pgmap v471: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:28.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:27 smithi082 ceph-mon[33535]: pgmap v471: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:28.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:27 smithi070 ceph-mon[33250]: pgmap v471: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:29.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:29 smithi067 ceph-mon[29731]: pgmap v472: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:30.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:29 smithi082 ceph-mon[33535]: pgmap v472: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:30.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:29 smithi070 ceph-mon[33250]: pgmap v472: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:31.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:31 smithi067 ceph-mon[29731]: pgmap v473: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:26:32.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:31 smithi082 ceph-mon[33535]: pgmap v473: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:26:32.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:31 smithi070 ceph-mon[33250]: pgmap v473: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:26:33.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:33 smithi067 ceph-mon[29731]: pgmap v474: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:34.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:33 smithi082 ceph-mon[33535]: pgmap v474: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:34.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:33 smithi070 ceph-mon[33250]: pgmap v474: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:35.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:35 smithi067 ceph-mon[29731]: pgmap v475: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:26:36.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:35 smithi082 ceph-mon[33535]: pgmap v475: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:26:36.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:35 smithi070 ceph-mon[33250]: pgmap v475: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:26:37.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:37 smithi067 ceph-mon[29731]: pgmap v476: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:38.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:37 smithi082 ceph-mon[33535]: pgmap v476: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:38.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:37 smithi070 ceph-mon[33250]: pgmap v476: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:39.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:39 smithi067 ceph-mon[29731]: pgmap v477: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:40.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:39 smithi082 ceph-mon[33535]: pgmap v477: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:40.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:39 smithi070 ceph-mon[33250]: pgmap v477: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:26:41.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:41 smithi067 ceph-mon[29731]: pgmap v478: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:26:42.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:41 smithi082 ceph-mon[33535]: pgmap v478: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:26:42.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:41 smithi070 ceph-mon[33250]: pgmap v478: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:26:43.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:43 smithi067 ceph-mon[29731]: pgmap v479: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-05-03T17:26:44.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:43 smithi082 ceph-mon[33535]: pgmap v479: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-05-03T17:26:44.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:43 smithi070 ceph-mon[33250]: pgmap v479: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-05-03T17:26:45.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:45 smithi067 ceph-mon[29731]: pgmap v480: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-03T17:26:46.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:45 smithi082 ceph-mon[33535]: pgmap v480: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-03T17:26:46.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:45 smithi070 ceph-mon[33250]: pgmap v480: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-03T17:26:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:47 smithi067 ceph-mon[29731]: pgmap v481: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T17:26:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:26:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:26:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:26:47.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:26:48.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:47 smithi082 ceph-mon[33535]: pgmap v481: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T17:26:48.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:47 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:26:48.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:47 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:26:48.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:47 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:26:48.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:47 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:26:48.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:47 smithi070 ceph-mon[33250]: pgmap v481: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T17:26:48.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:47 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:26:48.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:47 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:26:48.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:47 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:26:48.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:47 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:26:49.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:49 smithi067 ceph-mon[29731]: pgmap v482: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:26:50.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:49 smithi082 ceph-mon[33535]: pgmap v482: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:26:50.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:49 smithi070 ceph-mon[33250]: pgmap v482: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:26:51.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:51 smithi067 ceph-mon[29731]: pgmap v483: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:26:52.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:51 smithi082 ceph-mon[33535]: pgmap v483: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:26:52.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:51 smithi070 ceph-mon[33250]: pgmap v483: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:26:53.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:53 smithi067 ceph-mon[29731]: pgmap v484: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:54.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:53 smithi082 ceph-mon[33535]: pgmap v484: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:54.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:53 smithi070 ceph-mon[33250]: pgmap v484: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:26:54.920 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:26:54.920+0000 7f57417fa640 1 -- 172.21.15.67:0/1530589646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f57340034c0 con 0x7f574c1034c0 2024-05-03T17:26:54.921 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:26:54.920+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 <== mon.0 v2:172.21.15.67:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f57340034c0 con 0x7f574c1034c0 2024-05-03T17:26:55.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:55 smithi067 ceph-mon[29731]: pgmap v485: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-03T17:26:56.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:55 smithi082 ceph-mon[33535]: pgmap v485: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-03T17:26:56.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:55 smithi070 ceph-mon[33250]: pgmap v485: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-03T17:26:57.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:57 smithi067 ceph-mon[29731]: pgmap v486: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:58.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:57 smithi082 ceph-mon[33535]: pgmap v486: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:58.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:57 smithi070 ceph-mon[33250]: pgmap v486: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:26:59.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:26:59 smithi067 ceph-mon[29731]: pgmap v487: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:00.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:26:59 smithi082 ceph-mon[33535]: pgmap v487: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:00.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:26:59 smithi070 ceph-mon[33250]: pgmap v487: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:01.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:01 smithi067 ceph-mon[29731]: pgmap v488: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:02.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:01 smithi082 ceph-mon[33535]: pgmap v488: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:02.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:01 smithi070 ceph-mon[33250]: pgmap v488: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:03.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:03 smithi067 ceph-mon[29731]: pgmap v489: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:27:04.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:03 smithi082 ceph-mon[33535]: pgmap v489: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:27:04.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:03 smithi070 ceph-mon[33250]: pgmap v489: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:27:05.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:05 smithi067 ceph-mon[29731]: pgmap v490: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:27:06.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:05 smithi082 ceph-mon[33535]: pgmap v490: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:27:06.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:05 smithi070 ceph-mon[33250]: pgmap v490: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:27:08.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:07 smithi082 ceph-mon[33535]: pgmap v491: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:08.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:07 smithi070 ceph-mon[33250]: pgmap v491: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:08.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:07 smithi067 ceph-mon[29731]: pgmap v491: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:10.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:09 smithi082 ceph-mon[33535]: pgmap v492: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:27:10.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:09 smithi070 ceph-mon[33250]: pgmap v492: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:27:10.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:09 smithi067 ceph-mon[29731]: pgmap v492: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:27:12.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:11 smithi082 ceph-mon[33535]: pgmap v493: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:12.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:11 smithi070 ceph-mon[33250]: pgmap v493: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:12.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:11 smithi067 ceph-mon[29731]: pgmap v493: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:14.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:13 smithi082 ceph-mon[33535]: pgmap v494: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:27:14.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:13 smithi070 ceph-mon[33250]: pgmap v494: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:27:14.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:13 smithi067 ceph-mon[29731]: pgmap v494: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-03T17:27:16.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:15 smithi082 ceph-mon[33535]: pgmap v495: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:27:16.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:15 smithi070 ceph-mon[33250]: pgmap v495: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:27:16.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:15 smithi067 ceph-mon[29731]: pgmap v495: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:27:18.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:17 smithi082 ceph-mon[33535]: pgmap v496: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:18.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:17 smithi070 ceph-mon[33250]: pgmap v496: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:18.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:17 smithi067 ceph-mon[29731]: pgmap v496: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:20.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:19 smithi082 ceph-mon[33535]: pgmap v497: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:27:20.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:19 smithi070 ceph-mon[33250]: pgmap v497: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:27:20.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:19 smithi067 ceph-mon[29731]: pgmap v497: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:27:22.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:21 smithi082 ceph-mon[33535]: pgmap v498: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:22.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:21 smithi070 ceph-mon[33250]: pgmap v498: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:22.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:21 smithi067 ceph-mon[29731]: pgmap v498: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:24.005 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:23 smithi067 ceph-mon[29731]: pgmap v499: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:24.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:23 smithi082 ceph-mon[33535]: pgmap v499: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:24.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:23 smithi070 ceph-mon[33250]: pgmap v499: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:26.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:25 smithi082 ceph-mon[33535]: pgmap v500: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:27:26.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:25 smithi070 ceph-mon[33250]: pgmap v500: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:27:26.206 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:25 smithi067 ceph-mon[29731]: pgmap v500: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-05-03T17:27:28.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:27 smithi082 ceph-mon[33535]: pgmap v501: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:28.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:27 smithi070 ceph-mon[33250]: pgmap v501: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:28.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:27 smithi067 ceph-mon[29731]: pgmap v501: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:30.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:29 smithi082 ceph-mon[33535]: pgmap v502: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:27:30.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:29 smithi070 ceph-mon[33250]: pgmap v502: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:27:30.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:29 smithi067 ceph-mon[29731]: pgmap v502: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:27:32.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:31 smithi082 ceph-mon[33535]: pgmap v503: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:32.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:31 smithi070 ceph-mon[33250]: pgmap v503: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:32.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:31 smithi067 ceph-mon[29731]: pgmap v503: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:34.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:33 smithi082 ceph-mon[33535]: pgmap v504: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:34.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:33 smithi070 ceph-mon[33250]: pgmap v504: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:34.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:33 smithi067 ceph-mon[29731]: pgmap v504: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:36.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:35 smithi082 ceph-mon[33535]: pgmap v505: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:36.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:35 smithi070 ceph-mon[33250]: pgmap v505: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:36.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:35 smithi067 ceph-mon[29731]: pgmap v505: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:38.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:37 smithi082 ceph-mon[33535]: pgmap v506: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:38.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:37 smithi070 ceph-mon[33250]: pgmap v506: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:38.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:37 smithi067 ceph-mon[29731]: pgmap v506: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:40.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:39 smithi082 ceph-mon[33535]: pgmap v507: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:40.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:39 smithi070 ceph-mon[33250]: pgmap v507: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:40.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:39 smithi067 ceph-mon[29731]: pgmap v507: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:42.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:41 smithi082 ceph-mon[33535]: pgmap v508: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:42.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:41 smithi070 ceph-mon[33250]: pgmap v508: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:42.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:41 smithi067 ceph-mon[29731]: pgmap v508: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:44.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:43 smithi082 ceph-mon[33535]: pgmap v509: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:44.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:43 smithi070 ceph-mon[33250]: pgmap v509: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:44.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:43 smithi067 ceph-mon[29731]: pgmap v509: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:46.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:45 smithi082 ceph-mon[33535]: pgmap v510: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:46.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:45 smithi070 ceph-mon[33250]: pgmap v510: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:46.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:45 smithi067 ceph-mon[29731]: pgmap v510: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:48.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:47 smithi082 ceph-mon[33535]: pgmap v511: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:48.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:47 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:27:48.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:47 smithi070 ceph-mon[33250]: pgmap v511: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:48.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:47 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:27:48.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:47 smithi067 ceph-mon[29731]: pgmap v511: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:48.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:47 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:27:49.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:48 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:27:49.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:48 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:27:49.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:48 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:27:49.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:48 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:27:49.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:48 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:27:49.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:48 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:27:49.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:48 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:27:49.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:48 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:27:49.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:48 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:27:50.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:49 smithi070 ceph-mon[33250]: pgmap v512: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:27:50.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:49 smithi067 ceph-mon[29731]: pgmap v512: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:27:50.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:49 smithi082 ceph-mon[33535]: pgmap v512: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:27:52.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:51 smithi070 ceph-mon[33250]: pgmap v513: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:52.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:51 smithi067 ceph-mon[29731]: pgmap v513: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:52.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:51 smithi082 ceph-mon[33535]: pgmap v513: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:54.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:53 smithi070 ceph-mon[33250]: pgmap v514: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:54.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:53 smithi067 ceph-mon[29731]: pgmap v514: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:54.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:53 smithi082 ceph-mon[33535]: pgmap v514: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:27:56.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:55 smithi070 ceph-mon[33250]: pgmap v515: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:56.066 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:55 smithi082 ceph-mon[33535]: pgmap v515: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:56.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:55 smithi067 ceph-mon[29731]: pgmap v515: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:27:58.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:57 smithi070 ceph-mon[33250]: pgmap v516: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:58.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:57 smithi067 ceph-mon[29731]: pgmap v516: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:27:58.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:57 smithi082 ceph-mon[33535]: pgmap v516: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:00.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:27:59 smithi070 ceph-mon[33250]: pgmap v517: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:00.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:27:59 smithi067 ceph-mon[29731]: pgmap v517: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:00.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:27:59 smithi082 ceph-mon[33535]: pgmap v517: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:02.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:01 smithi070 ceph-mon[33250]: pgmap v518: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:02.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:01 smithi067 ceph-mon[29731]: pgmap v518: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:02.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:01 smithi082 ceph-mon[33535]: pgmap v518: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:04.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:03 smithi070 ceph-mon[33250]: pgmap v519: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:04.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:03 smithi067 ceph-mon[29731]: pgmap v519: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:04.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:03 smithi082 ceph-mon[33535]: pgmap v519: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:06.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:05 smithi070 ceph-mon[33250]: pgmap v520: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:06.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:05 smithi067 ceph-mon[29731]: pgmap v520: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:06.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:05 smithi082 ceph-mon[33535]: pgmap v520: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:08.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:07 smithi070 ceph-mon[33250]: pgmap v521: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:08.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:07 smithi067 ceph-mon[29731]: pgmap v521: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:08.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:07 smithi082 ceph-mon[33535]: pgmap v521: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:10.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:09 smithi070 ceph-mon[33250]: pgmap v522: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:10.203 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:09 smithi067 ceph-mon[29731]: pgmap v522: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:10.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:09 smithi082 ceph-mon[33535]: pgmap v522: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:12.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:11 smithi070 ceph-mon[33250]: pgmap v523: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:12.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:11 smithi067 ceph-mon[29731]: pgmap v523: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:12.264 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:11 smithi082 ceph-mon[33535]: pgmap v523: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:14.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:14 smithi067 ceph-mon[29731]: pgmap v524: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:14.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:14 smithi082 ceph-mon[33535]: pgmap v524: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:14.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:14 smithi070 ceph-mon[33250]: pgmap v524: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:16.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:16 smithi067 ceph-mon[29731]: pgmap v525: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:16.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:16 smithi082 ceph-mon[33535]: pgmap v525: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:16.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:16 smithi070 ceph-mon[33250]: pgmap v525: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:18.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:18 smithi067 ceph-mon[29731]: pgmap v526: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:18.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:18 smithi082 ceph-mon[33535]: pgmap v526: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:18.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:18 smithi070 ceph-mon[33250]: pgmap v526: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:20.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:20 smithi067 ceph-mon[29731]: pgmap v527: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:20.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:20 smithi082 ceph-mon[33535]: pgmap v527: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:20.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:20 smithi070 ceph-mon[33250]: pgmap v527: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:22.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:22 smithi067 ceph-mon[29731]: pgmap v528: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:22.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:22 smithi082 ceph-mon[33535]: pgmap v528: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:22.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:22 smithi070 ceph-mon[33250]: pgmap v528: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:24.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:24 smithi067 ceph-mon[29731]: pgmap v529: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:24.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:24 smithi082 ceph-mon[33535]: pgmap v529: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:24.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:24 smithi070 ceph-mon[33250]: pgmap v529: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:26.362 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:26 smithi070 ceph-mon[33250]: pgmap v530: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:26.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:26 smithi067 ceph-mon[29731]: pgmap v530: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:26.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:26 smithi082 ceph-mon[33535]: pgmap v530: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:28.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:28 smithi067 ceph-mon[29731]: pgmap v531: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:28.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:28 smithi082 ceph-mon[33535]: pgmap v531: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:28.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:28 smithi070 ceph-mon[33250]: pgmap v531: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:30.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:30 smithi067 ceph-mon[29731]: pgmap v532: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:30.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:30 smithi082 ceph-mon[33535]: pgmap v532: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:30.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:30 smithi070 ceph-mon[33250]: pgmap v532: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:32.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:32 smithi067 ceph-mon[29731]: pgmap v533: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:32.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:32 smithi082 ceph-mon[33535]: pgmap v533: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:32.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:32 smithi070 ceph-mon[33250]: pgmap v533: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:34.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:34 smithi067 ceph-mon[29731]: pgmap v534: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:34.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:34 smithi082 ceph-mon[33535]: pgmap v534: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:34.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:34 smithi070 ceph-mon[33250]: pgmap v534: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:36.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:36 smithi067 ceph-mon[29731]: pgmap v535: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:36.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:36 smithi082 ceph-mon[33535]: pgmap v535: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:36.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:36 smithi070 ceph-mon[33250]: pgmap v535: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:38.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:38 smithi067 ceph-mon[29731]: pgmap v536: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:38.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:38 smithi082 ceph-mon[33535]: pgmap v536: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:38.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:38 smithi070 ceph-mon[33250]: pgmap v536: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:40.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:40 smithi067 ceph-mon[29731]: pgmap v537: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:40.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:40 smithi082 ceph-mon[33535]: pgmap v537: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:40.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:40 smithi070 ceph-mon[33250]: pgmap v537: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:42.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:42 smithi067 ceph-mon[29731]: pgmap v538: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:42.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:42 smithi082 ceph-mon[33535]: pgmap v538: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:42.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:42 smithi070 ceph-mon[33250]: pgmap v538: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:44.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:44 smithi067 ceph-mon[29731]: pgmap v539: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:44.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:44 smithi082 ceph-mon[33535]: pgmap v539: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:44.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:44 smithi070 ceph-mon[33250]: pgmap v539: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:46.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:46 smithi067 ceph-mon[29731]: pgmap v540: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:46.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:46 smithi082 ceph-mon[33535]: pgmap v540: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:46.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:46 smithi070 ceph-mon[33250]: pgmap v540: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:48.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:48 smithi067 ceph-mon[29731]: pgmap v541: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:48.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:48 smithi082 ceph-mon[33535]: pgmap v541: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:48.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:48 smithi070 ceph-mon[33250]: pgmap v541: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:49.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:28:49.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:28:49.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:28:49.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:28:49.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:28:49.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:28:49.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:28:49.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:28:49.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:28:49.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:28:49.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:28:49.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:28:49.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:28:49.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:28:49.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:28:49.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:28:49.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:28:49.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:28:49.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:28:49.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:28:49.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:28:49.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:28:49.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:28:49.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:28:50.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:50 smithi067 ceph-mon[29731]: pgmap v542: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:50.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:50 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi067 to 176.2M 2024-05-03T17:28:50.453 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:50 smithi067 ceph-mon[29731]: Unable to set osd_memory_target on smithi067 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-05-03T17:28:50.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:50 smithi082 ceph-mon[33535]: pgmap v542: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:50.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:50 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi067 to 176.2M 2024-05-03T17:28:50.514 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:50 smithi082 ceph-mon[33535]: Unable to set osd_memory_target on smithi067 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-05-03T17:28:50.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:50 smithi070 ceph-mon[33250]: pgmap v542: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:28:50.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:50 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi067 to 176.2M 2024-05-03T17:28:50.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:50 smithi070 ceph-mon[33250]: Unable to set osd_memory_target on smithi067 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-05-03T17:28:52.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:52 smithi067 ceph-mon[29731]: pgmap v543: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:52.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:52 smithi082 ceph-mon[33535]: pgmap v543: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:52.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:52 smithi070 ceph-mon[33250]: pgmap v543: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:54.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:54 smithi067 ceph-mon[29731]: pgmap v544: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:54.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:54 smithi082 ceph-mon[33535]: pgmap v544: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:54.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:54 smithi070 ceph-mon[33250]: pgmap v544: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:28:56.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:56 smithi067 ceph-mon[29731]: pgmap v545: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:56.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:56 smithi082 ceph-mon[33535]: pgmap v545: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:56.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:56 smithi070 ceph-mon[33250]: pgmap v545: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:28:58.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:28:58 smithi067 ceph-mon[29731]: pgmap v546: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:58.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:28:58 smithi082 ceph-mon[33535]: pgmap v546: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:28:58.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:28:58 smithi070 ceph-mon[33250]: pgmap v546: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:00.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:00 smithi067 ceph-mon[29731]: pgmap v547: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:00.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:00 smithi082 ceph-mon[33535]: pgmap v547: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:00.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:00 smithi070 ceph-mon[33250]: pgmap v547: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:02.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:02 smithi067 ceph-mon[29731]: pgmap v548: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:02.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:02 smithi082 ceph-mon[33535]: pgmap v548: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:02.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:02 smithi070 ceph-mon[33250]: pgmap v548: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:04.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:04 smithi067 ceph-mon[29731]: pgmap v549: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:04.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:04 smithi082 ceph-mon[33535]: pgmap v549: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:04.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:04 smithi070 ceph-mon[33250]: pgmap v549: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:06.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:06 smithi067 ceph-mon[29731]: pgmap v550: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:06.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:06 smithi082 ceph-mon[33535]: pgmap v550: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:06.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:06 smithi070 ceph-mon[33250]: pgmap v550: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:07.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:07 smithi067 ceph-mon[29731]: pgmap v551: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:07.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:07 smithi082 ceph-mon[33535]: pgmap v551: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:07.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:07 smithi070 ceph-mon[33250]: pgmap v551: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:09.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:09 smithi067 ceph-mon[29731]: pgmap v552: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:10.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:09 smithi082 ceph-mon[33535]: pgmap v552: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:10.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:09 smithi070 ceph-mon[33250]: pgmap v552: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:11.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:11 smithi067 ceph-mon[29731]: pgmap v553: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:12.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:11 smithi082 ceph-mon[33535]: pgmap v553: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:12.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:11 smithi070 ceph-mon[33250]: pgmap v553: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:13.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:13 smithi067 ceph-mon[29731]: pgmap v554: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:14.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:13 smithi082 ceph-mon[33535]: pgmap v554: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:14.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:13 smithi070 ceph-mon[33250]: pgmap v554: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:15.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:15 smithi067 ceph-mon[29731]: pgmap v555: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:16.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:15 smithi082 ceph-mon[33535]: pgmap v555: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:16.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:15 smithi070 ceph-mon[33250]: pgmap v555: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:17.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:17 smithi067 ceph-mon[29731]: pgmap v556: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:18.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:17 smithi082 ceph-mon[33535]: pgmap v556: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:18.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:17 smithi070 ceph-mon[33250]: pgmap v556: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:19.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:19 smithi067 ceph-mon[29731]: pgmap v557: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:20.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:19 smithi082 ceph-mon[33535]: pgmap v557: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:20.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:19 smithi070 ceph-mon[33250]: pgmap v557: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:21.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:21 smithi067 ceph-mon[29731]: pgmap v558: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:22.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:21 smithi082 ceph-mon[33535]: pgmap v558: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:22.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:21 smithi070 ceph-mon[33250]: pgmap v558: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:23.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:23 smithi067 ceph-mon[29731]: pgmap v559: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:24.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:23 smithi082 ceph-mon[33535]: pgmap v559: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:24.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:23 smithi070 ceph-mon[33250]: pgmap v559: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:25.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:25 smithi067 ceph-mon[29731]: pgmap v560: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:26.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:25 smithi082 ceph-mon[33535]: pgmap v560: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:26.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:25 smithi070 ceph-mon[33250]: pgmap v560: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:27.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:27 smithi067 ceph-mon[29731]: pgmap v561: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:28.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:27 smithi082 ceph-mon[33535]: pgmap v561: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:28.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:27 smithi070 ceph-mon[33250]: pgmap v561: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:29.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:29 smithi067 ceph-mon[29731]: pgmap v562: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:30.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:29 smithi082 ceph-mon[33535]: pgmap v562: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:30.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:29 smithi070 ceph-mon[33250]: pgmap v562: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:31.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:31 smithi067 ceph-mon[29731]: pgmap v563: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:32.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:31 smithi082 ceph-mon[33535]: pgmap v563: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:32.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:31 smithi070 ceph-mon[33250]: pgmap v563: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:33.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:33 smithi067 ceph-mon[29731]: pgmap v564: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:34.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:33 smithi082 ceph-mon[33535]: pgmap v564: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:34.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:33 smithi070 ceph-mon[33250]: pgmap v564: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:35.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:35 smithi067 ceph-mon[29731]: pgmap v565: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:36.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:35 smithi082 ceph-mon[33535]: pgmap v565: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:36.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:35 smithi070 ceph-mon[33250]: pgmap v565: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:37.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:37 smithi067 ceph-mon[29731]: pgmap v566: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:38.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:37 smithi082 ceph-mon[33535]: pgmap v566: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:38.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:37 smithi070 ceph-mon[33250]: pgmap v566: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:39.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:39 smithi067 ceph-mon[29731]: pgmap v567: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:40.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:39 smithi082 ceph-mon[33535]: pgmap v567: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:40.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:39 smithi070 ceph-mon[33250]: pgmap v567: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:41.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:41 smithi067 ceph-mon[29731]: pgmap v568: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:42.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:41 smithi082 ceph-mon[33535]: pgmap v568: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:42.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:41 smithi070 ceph-mon[33250]: pgmap v568: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:43.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:43 smithi067 ceph-mon[29731]: pgmap v569: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:44.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:43 smithi082 ceph-mon[33535]: pgmap v569: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:44.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:43 smithi070 ceph-mon[33250]: pgmap v569: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:45.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:45 smithi067 ceph-mon[29731]: pgmap v570: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:46.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:45 smithi082 ceph-mon[33535]: pgmap v570: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:46.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:45 smithi070 ceph-mon[33250]: pgmap v570: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:47.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:47 smithi067 ceph-mon[29731]: pgmap v571: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:48.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:47 smithi082 ceph-mon[33535]: pgmap v571: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:48.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:47 smithi070 ceph-mon[33250]: pgmap v571: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:49.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:49 smithi067 ceph-mon[29731]: pgmap v572: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:49.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:49 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:29:50.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:49 smithi082 ceph-mon[33535]: pgmap v572: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:50.015 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:49 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:29:50.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:49 smithi070 ceph-mon[33250]: pgmap v572: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:29:50.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:49 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:29:50.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:50.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:50.952 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:50.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:50.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:50.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:50.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:50.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:50.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-05-03T17:29:50.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi082 to 176.2M 2024-05-03T17:29:50.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: Unable to set osd_memory_target on smithi070 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-05-03T17:29:50.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: Unable to set osd_memory_target on smithi082 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-05-03T17:29:50.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:29:50.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:29:50.953 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:29:51.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi082 to 176.2M 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: Unable to set osd_memory_target on smithi070 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: Unable to set osd_memory_target on smithi082 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:29:51.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:29:51.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:29:51.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi070 to 176.2M 2024-05-03T17:29:51.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi082 to 176.2M 2024-05-03T17:29:51.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: Unable to set osd_memory_target on smithi070 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-05-03T17:29:51.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: Unable to set osd_memory_target on smithi082 to 184763801: error parsing value: Value '184763801' is below minimum 939524096 2024-05-03T17:29:51.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:29:51.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:29:51.058 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:29:52.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:51 smithi067 ceph-mon[29731]: pgmap v573: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:52.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:51 smithi082 ceph-mon[33535]: pgmap v573: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:52.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:51 smithi070 ceph-mon[33250]: pgmap v573: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:54.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:53 smithi067 ceph-mon[29731]: pgmap v574: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:54.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:53 smithi082 ceph-mon[33535]: pgmap v574: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:54.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:53 smithi070 ceph-mon[33250]: pgmap v574: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:29:54.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:29:54.946+0000 7f57417fa640 1 -- 172.21.15.67:0/1530589646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f5734005820 con 0x7f574c1034c0 2024-05-03T17:29:54.945 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:29:54.946+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 <== mon.0 v2:172.21.15.67:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f5734005820 con 0x7f574c1034c0 2024-05-03T17:29:56.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:55 smithi067 ceph-mon[29731]: pgmap v575: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:56.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:55 smithi082 ceph-mon[33535]: pgmap v575: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:56.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:55 smithi070 ceph-mon[33250]: pgmap v575: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:29:58.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:57 smithi067 ceph-mon[29731]: pgmap v576: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:58.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:57 smithi082 ceph-mon[33535]: pgmap v576: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:29:58.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:57 smithi070 ceph-mon[33250]: pgmap v576: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:00.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:29:59 smithi067 ceph-mon[29731]: pgmap v577: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:00.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:29:59 smithi082 ceph-mon[33535]: pgmap v577: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:00.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:29:59 smithi070 ceph-mon[33250]: pgmap v577: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:01.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:00 smithi067 ceph-mon[29731]: overall HEALTH_OK 2024-05-03T17:30:01.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:00 smithi082 ceph-mon[33535]: overall HEALTH_OK 2024-05-03T17:30:01.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:00 smithi070 ceph-mon[33250]: overall HEALTH_OK 2024-05-03T17:30:02.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:01 smithi067 ceph-mon[29731]: pgmap v578: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:02.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:01 smithi082 ceph-mon[33535]: pgmap v578: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:02.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:01 smithi070 ceph-mon[33250]: pgmap v578: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:04.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:03 smithi067 ceph-mon[29731]: pgmap v579: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:04.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:03 smithi082 ceph-mon[33535]: pgmap v579: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:04.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:03 smithi070 ceph-mon[33250]: pgmap v579: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:06.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:05 smithi067 ceph-mon[29731]: pgmap v580: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:06.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:05 smithi082 ceph-mon[33535]: pgmap v580: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:06.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:05 smithi070 ceph-mon[33250]: pgmap v580: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:08.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:07 smithi067 ceph-mon[29731]: pgmap v581: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:08.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:07 smithi082 ceph-mon[33535]: pgmap v581: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:08.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:07 smithi070 ceph-mon[33250]: pgmap v581: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:10.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:09 smithi067 ceph-mon[29731]: pgmap v582: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:10.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:09 smithi082 ceph-mon[33535]: pgmap v582: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:10.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:09 smithi070 ceph-mon[33250]: pgmap v582: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:12.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:11 smithi067 ceph-mon[29731]: pgmap v583: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:12.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:11 smithi082 ceph-mon[33535]: pgmap v583: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:12.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:11 smithi070 ceph-mon[33250]: pgmap v583: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:14.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:13 smithi067 ceph-mon[29731]: pgmap v584: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:14.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:13 smithi082 ceph-mon[33535]: pgmap v584: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:14.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:13 smithi070 ceph-mon[33250]: pgmap v584: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:16.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:15 smithi067 ceph-mon[29731]: pgmap v585: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:16.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:15 smithi082 ceph-mon[33535]: pgmap v585: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:16.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:15 smithi070 ceph-mon[33250]: pgmap v585: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:18.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:18 smithi067 ceph-mon[29731]: pgmap v586: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:18.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:18 smithi082 ceph-mon[33535]: pgmap v586: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:18.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:18 smithi070 ceph-mon[33250]: pgmap v586: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:20.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:20 smithi067 ceph-mon[29731]: pgmap v587: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:20.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:20 smithi082 ceph-mon[33535]: pgmap v587: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:20.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:20 smithi070 ceph-mon[33250]: pgmap v587: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:22.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:22 smithi067 ceph-mon[29731]: pgmap v588: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:22.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:22 smithi082 ceph-mon[33535]: pgmap v588: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:22.557 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:22 smithi070 ceph-mon[33250]: pgmap v588: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:24.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:24 smithi067 ceph-mon[29731]: pgmap v589: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:24.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:24 smithi082 ceph-mon[33535]: pgmap v589: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:24.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:24 smithi070 ceph-mon[33250]: pgmap v589: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:26.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:26 smithi067 ceph-mon[29731]: pgmap v590: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:26.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:26 smithi082 ceph-mon[33535]: pgmap v590: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:26.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:26 smithi070 ceph-mon[33250]: pgmap v590: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:28.377 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:28 smithi070 ceph-mon[33250]: pgmap v591: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:28.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:28 smithi067 ceph-mon[29731]: pgmap v591: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:28.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:28 smithi082 ceph-mon[33535]: pgmap v591: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:30.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:30 smithi067 ceph-mon[29731]: pgmap v592: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:30.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:30 smithi082 ceph-mon[33535]: pgmap v592: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:30.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:30 smithi070 ceph-mon[33250]: pgmap v592: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:32.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:32 smithi067 ceph-mon[29731]: pgmap v593: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:32.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:32 smithi082 ceph-mon[33535]: pgmap v593: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:32.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:32 smithi070 ceph-mon[33250]: pgmap v593: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:34.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:34 smithi067 ceph-mon[29731]: pgmap v594: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:34.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:34 smithi082 ceph-mon[33535]: pgmap v594: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:34.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:34 smithi070 ceph-mon[33250]: pgmap v594: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:36.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:36 smithi067 ceph-mon[29731]: pgmap v595: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:36.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:36 smithi082 ceph-mon[33535]: pgmap v595: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:36.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:36 smithi070 ceph-mon[33250]: pgmap v595: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:38.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:38 smithi067 ceph-mon[29731]: pgmap v596: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:38.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:38 smithi082 ceph-mon[33535]: pgmap v596: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:38.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:38 smithi070 ceph-mon[33250]: pgmap v596: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:40.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:40 smithi067 ceph-mon[29731]: pgmap v597: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:40.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:40 smithi082 ceph-mon[33535]: pgmap v597: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:40.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:40 smithi070 ceph-mon[33250]: pgmap v597: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:42.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:42 smithi067 ceph-mon[29731]: pgmap v598: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:42.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:42 smithi082 ceph-mon[33535]: pgmap v598: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:42.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:42 smithi070 ceph-mon[33250]: pgmap v598: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:44.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:44 smithi067 ceph-mon[29731]: pgmap v599: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:44.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:44 smithi082 ceph-mon[33535]: pgmap v599: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:44.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:44 smithi070 ceph-mon[33250]: pgmap v599: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:46.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:46 smithi067 ceph-mon[29731]: pgmap v600: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:46.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:46 smithi082 ceph-mon[33535]: pgmap v600: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:46.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:46 smithi070 ceph-mon[33250]: pgmap v600: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:48.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:48 smithi067 ceph-mon[29731]: pgmap v601: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:48.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:48 smithi082 ceph-mon[33535]: pgmap v601: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:48.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:48 smithi070 ceph-mon[33250]: pgmap v601: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:50.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:50 smithi067 ceph-mon[29731]: pgmap v602: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:50.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:50 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:30:50.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:50 smithi082 ceph-mon[33535]: pgmap v602: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:50.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:50 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:30:50.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:50 smithi070 ceph-mon[33250]: pgmap v602: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:30:50.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:50 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:30:51.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:51 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:30:51.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:51 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:30:51.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:51 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:30:51.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:51 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:30:51.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:51 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:30:51.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:51 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:30:51.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:51 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:30:51.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:51 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:30:51.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:51 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:30:52.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:52 smithi067 ceph-mon[29731]: pgmap v603: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:52.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:52 smithi082 ceph-mon[33535]: pgmap v603: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:52.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:52 smithi070 ceph-mon[33250]: pgmap v603: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:54.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:54 smithi067 ceph-mon[29731]: pgmap v604: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:54.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:54 smithi082 ceph-mon[33535]: pgmap v604: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:54.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:54 smithi070 ceph-mon[33250]: pgmap v604: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:30:56.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:56 smithi067 ceph-mon[29731]: pgmap v605: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:56.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:56 smithi082 ceph-mon[33535]: pgmap v605: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:56.558 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:56 smithi070 ceph-mon[33250]: pgmap v605: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:30:58.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:30:58 smithi067 ceph-mon[29731]: pgmap v606: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:58.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:30:58 smithi082 ceph-mon[33535]: pgmap v606: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:30:58.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:30:58 smithi070 ceph-mon[33250]: pgmap v606: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:00.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:00 smithi067 ceph-mon[29731]: pgmap v607: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:00.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:00 smithi082 ceph-mon[33535]: pgmap v607: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:00.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:00 smithi070 ceph-mon[33250]: pgmap v607: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:02.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:02 smithi067 ceph-mon[29731]: pgmap v608: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:02.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:02 smithi082 ceph-mon[33535]: pgmap v608: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:02.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:02 smithi070 ceph-mon[33250]: pgmap v608: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:04.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:04 smithi067 ceph-mon[29731]: pgmap v609: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:04.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:04 smithi082 ceph-mon[33535]: pgmap v609: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:04.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:04 smithi070 ceph-mon[33250]: pgmap v609: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:06.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:06 smithi067 ceph-mon[29731]: pgmap v610: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:06.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:06 smithi082 ceph-mon[33535]: pgmap v610: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:06.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:06 smithi070 ceph-mon[33250]: pgmap v610: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:07.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:07 smithi082 ceph-mon[33535]: pgmap v611: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:07.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:07 smithi070 ceph-mon[33250]: pgmap v611: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:07.702 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:07 smithi067 ceph-mon[29731]: pgmap v611: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:09.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:09 smithi067 ceph-mon[29731]: pgmap v612: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:10.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:09 smithi082 ceph-mon[33535]: pgmap v612: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:10.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:09 smithi070 ceph-mon[33250]: pgmap v612: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:11.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:11 smithi067 ceph-mon[29731]: pgmap v613: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:12.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:11 smithi082 ceph-mon[33535]: pgmap v613: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:12.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:11 smithi070 ceph-mon[33250]: pgmap v613: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:13.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:13 smithi067 ceph-mon[29731]: pgmap v614: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:14.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:13 smithi082 ceph-mon[33535]: pgmap v614: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:14.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:13 smithi070 ceph-mon[33250]: pgmap v614: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:15.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:15 smithi067 ceph-mon[29731]: pgmap v615: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:16.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:15 smithi082 ceph-mon[33535]: pgmap v615: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:16.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:15 smithi070 ceph-mon[33250]: pgmap v615: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:17.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:17 smithi067 ceph-mon[29731]: pgmap v616: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:18.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:17 smithi082 ceph-mon[33535]: pgmap v616: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:18.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:17 smithi070 ceph-mon[33250]: pgmap v616: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:19.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:19 smithi067 ceph-mon[29731]: pgmap v617: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:20.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:19 smithi082 ceph-mon[33535]: pgmap v617: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:20.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:19 smithi070 ceph-mon[33250]: pgmap v617: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:21.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:21 smithi067 ceph-mon[29731]: pgmap v618: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:22.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:21 smithi082 ceph-mon[33535]: pgmap v618: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:22.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:21 smithi070 ceph-mon[33250]: pgmap v618: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:23.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:23 smithi067 ceph-mon[29731]: pgmap v619: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:24.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:23 smithi082 ceph-mon[33535]: pgmap v619: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:24.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:23 smithi070 ceph-mon[33250]: pgmap v619: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:25.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:25 smithi067 ceph-mon[29731]: pgmap v620: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:26.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:25 smithi082 ceph-mon[33535]: pgmap v620: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:26.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:25 smithi070 ceph-mon[33250]: pgmap v620: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:27.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:27 smithi067 ceph-mon[29731]: pgmap v621: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:28.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:27 smithi082 ceph-mon[33535]: pgmap v621: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:28.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:27 smithi070 ceph-mon[33250]: pgmap v621: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:29.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:29 smithi067 ceph-mon[29731]: pgmap v622: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:30.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:29 smithi082 ceph-mon[33535]: pgmap v622: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:30.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:29 smithi070 ceph-mon[33250]: pgmap v622: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:31.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:31 smithi067 ceph-mon[29731]: pgmap v623: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:32.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:31 smithi082 ceph-mon[33535]: pgmap v623: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:32.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:31 smithi070 ceph-mon[33250]: pgmap v623: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:33.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:33 smithi067 ceph-mon[29731]: pgmap v624: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:34.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:33 smithi082 ceph-mon[33535]: pgmap v624: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:34.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:33 smithi070 ceph-mon[33250]: pgmap v624: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:35.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:35 smithi067 ceph-mon[29731]: pgmap v625: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:36.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:35 smithi082 ceph-mon[33535]: pgmap v625: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:36.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:35 smithi070 ceph-mon[33250]: pgmap v625: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:37.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:37 smithi067 ceph-mon[29731]: pgmap v626: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:38.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:37 smithi082 ceph-mon[33535]: pgmap v626: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:38.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:37 smithi070 ceph-mon[33250]: pgmap v626: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:39.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:39 smithi067 ceph-mon[29731]: pgmap v627: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:40.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:39 smithi082 ceph-mon[33535]: pgmap v627: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:40.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:39 smithi070 ceph-mon[33250]: pgmap v627: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:41.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:41 smithi067 ceph-mon[29731]: pgmap v628: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:42.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:41 smithi082 ceph-mon[33535]: pgmap v628: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:42.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:41 smithi070 ceph-mon[33250]: pgmap v628: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:43.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:43 smithi067 ceph-mon[29731]: pgmap v629: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:44.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:43 smithi082 ceph-mon[33535]: pgmap v629: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:44.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:43 smithi070 ceph-mon[33250]: pgmap v629: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:46.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:45 smithi082 ceph-mon[33535]: pgmap v630: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:46.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:45 smithi070 ceph-mon[33250]: pgmap v630: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:46.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:45 smithi067 ceph-mon[29731]: pgmap v630: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:48.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:47 smithi082 ceph-mon[33535]: pgmap v631: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:48.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:47 smithi070 ceph-mon[33250]: pgmap v631: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:48.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:47 smithi067 ceph-mon[29731]: pgmap v631: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:50.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:49 smithi082 ceph-mon[33535]: pgmap v632: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:50.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:49 smithi070 ceph-mon[33250]: pgmap v632: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:50.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:49 smithi067 ceph-mon[29731]: pgmap v632: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:31:52.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:51 smithi082 ceph-mon[33535]: pgmap v633: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:52.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:51 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:31:52.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:51 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:31:52.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:51 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:31:52.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:51 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:31:52.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:51 smithi070 ceph-mon[33250]: pgmap v633: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:52.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:51 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:31:52.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:51 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:31:52.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:51 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:31:52.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:51 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:31:52.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:51 smithi067 ceph-mon[29731]: pgmap v633: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:52.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:51 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:31:52.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:51 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:31:52.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:51 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:31:52.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:51 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:31:54.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:53 smithi082 ceph-mon[33535]: pgmap v634: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:54.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:53 smithi070 ceph-mon[33250]: pgmap v634: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:54.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:53 smithi067 ceph-mon[29731]: pgmap v634: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:31:56.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:55 smithi082 ceph-mon[33535]: pgmap v635: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:56.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:55 smithi070 ceph-mon[33250]: pgmap v635: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:56.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:55 smithi067 ceph-mon[29731]: pgmap v635: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:31:58.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:57 smithi067 ceph-mon[29731]: pgmap v636: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:58.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:57 smithi082 ceph-mon[33535]: pgmap v636: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:31:58.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:57 smithi070 ceph-mon[33250]: pgmap v636: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:00.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:31:59 smithi082 ceph-mon[33535]: pgmap v637: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:32:00.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:31:59 smithi070 ceph-mon[33250]: pgmap v637: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:32:00.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:31:59 smithi067 ceph-mon[29731]: pgmap v637: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:32:02.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:01 smithi082 ceph-mon[33535]: pgmap v638: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:02.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:01 smithi070 ceph-mon[33250]: pgmap v638: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:02.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:01 smithi067 ceph-mon[29731]: pgmap v638: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:04.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:03 smithi082 ceph-mon[33535]: pgmap v639: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:04.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:03 smithi070 ceph-mon[33250]: pgmap v639: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:04.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:03 smithi067 ceph-mon[29731]: pgmap v639: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:06.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:05 smithi082 ceph-mon[33535]: pgmap v640: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:06.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:05 smithi070 ceph-mon[33250]: pgmap v640: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:06.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:05 smithi067 ceph-mon[29731]: pgmap v640: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:08.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:07 smithi082 ceph-mon[33535]: pgmap v641: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:08.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:07 smithi070 ceph-mon[33250]: pgmap v641: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:08.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:07 smithi067 ceph-mon[29731]: pgmap v641: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:32:09.965+0000 7f57417fa640 1 -- 172.21.15.67:0/1530589646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- auth(proto 2 132 bytes epoch 0) -- 0x7f5734005b60 con 0x7f574c1034c0 2024-05-03T17:32:09.964 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:32:09.965+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 <== mon.0 v2:172.21.15.67:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (secure 0 0 0) 0x7f5734005b60 con 0x7f574c1034c0 2024-05-03T17:32:10.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:09 smithi082 ceph-mon[33535]: pgmap v642: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:32:10.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:09 smithi070 ceph-mon[33250]: pgmap v642: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:32:10.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:09 smithi067 ceph-mon[29731]: pgmap v642: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:32:12.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:11 smithi082 ceph-mon[33535]: pgmap v643: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:12.056 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:11 smithi070 ceph-mon[33250]: pgmap v643: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:12.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:11 smithi067 ceph-mon[29731]: pgmap v643: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:14.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:14 smithi067 ceph-mon[29731]: pgmap v644: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:14.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:14 smithi082 ceph-mon[33535]: pgmap v644: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:14.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:14 smithi070 ceph-mon[33250]: pgmap v644: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:16.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:16 smithi067 ceph-mon[29731]: pgmap v645: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:16.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:16 smithi082 ceph-mon[33535]: pgmap v645: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:16.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:16 smithi070 ceph-mon[33250]: pgmap v645: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:18.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:18 smithi067 ceph-mon[29731]: pgmap v646: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:18.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:18 smithi082 ceph-mon[33535]: pgmap v646: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:18.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:18 smithi070 ceph-mon[33250]: pgmap v646: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:20.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:20 smithi067 ceph-mon[29731]: pgmap v647: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:20.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:20 smithi082 ceph-mon[33535]: pgmap v647: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:20.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:20 smithi070 ceph-mon[33250]: pgmap v647: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:22.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:22 smithi067 ceph-mon[29731]: pgmap v648: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:22.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:22 smithi082 ceph-mon[33535]: pgmap v648: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:22.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:22 smithi070 ceph-mon[33250]: pgmap v648: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:24.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:24 smithi067 ceph-mon[29731]: pgmap v649: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:32:24.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:24 smithi082 ceph-mon[33535]: pgmap v649: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:32:24.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:24 smithi070 ceph-mon[33250]: pgmap v649: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:32:26.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:26 smithi067 ceph-mon[29731]: pgmap v650: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T17:32:26.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:26 smithi082 ceph-mon[33535]: pgmap v650: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T17:32:26.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:26 smithi070 ceph-mon[33250]: pgmap v650: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T17:32:28.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:28 smithi067 ceph-mon[29731]: pgmap v651: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-05-03T17:32:28.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:28 smithi082 ceph-mon[33535]: pgmap v651: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-05-03T17:32:28.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:28 smithi070 ceph-mon[33250]: pgmap v651: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 4 op/s 2024-05-03T17:32:30.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:30 smithi067 ceph-mon[29731]: pgmap v652: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:32:30.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:30 smithi082 ceph-mon[33535]: pgmap v652: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:32:30.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:30 smithi070 ceph-mon[33250]: pgmap v652: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:32:32.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:32 smithi067 ceph-mon[29731]: pgmap v653: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T17:32:32.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:32 smithi082 ceph-mon[33535]: pgmap v653: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T17:32:32.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:32 smithi070 ceph-mon[33250]: pgmap v653: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-03T17:32:34.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:34 smithi067 ceph-mon[29731]: pgmap v654: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:32:34.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:34 smithi082 ceph-mon[33535]: pgmap v654: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:32:34.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:34 smithi070 ceph-mon[33250]: pgmap v654: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:32:36.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:36 smithi067 ceph-mon[29731]: pgmap v655: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:36.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:36 smithi082 ceph-mon[33535]: pgmap v655: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:36.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:36 smithi070 ceph-mon[33250]: pgmap v655: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:38.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:38 smithi067 ceph-mon[29731]: pgmap v656: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:38.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:38 smithi082 ceph-mon[33535]: pgmap v656: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:38.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:38 smithi070 ceph-mon[33250]: pgmap v656: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:40.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:40 smithi067 ceph-mon[29731]: pgmap v657: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:32:40.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:40 smithi082 ceph-mon[33535]: pgmap v657: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:32:40.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:40 smithi070 ceph-mon[33250]: pgmap v657: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:32:42.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:42 smithi067 ceph-mon[29731]: pgmap v658: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:42.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:42 smithi082 ceph-mon[33535]: pgmap v658: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:42.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:42 smithi070 ceph-mon[33250]: pgmap v658: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:44.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:44 smithi067 ceph-mon[29731]: pgmap v659: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:44.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:44 smithi082 ceph-mon[33535]: pgmap v659: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:44.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:44 smithi070 ceph-mon[33250]: pgmap v659: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:46.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:46 smithi067 ceph-mon[29731]: pgmap v660: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:46.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:46 smithi082 ceph-mon[33535]: pgmap v660: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:46.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:46 smithi070 ceph-mon[33250]: pgmap v660: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:48.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:48 smithi067 ceph-mon[29731]: pgmap v661: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:48.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:48 smithi082 ceph-mon[33535]: pgmap v661: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:48.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:48 smithi070 ceph-mon[33250]: pgmap v661: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:50.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:50 smithi067 ceph-mon[29731]: pgmap v662: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:32:50.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:50 smithi082 ceph-mon[33535]: pgmap v662: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:32:50.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:50 smithi070 ceph-mon[33250]: pgmap v662: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:32:52.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:52 smithi067 ceph-mon[29731]: pgmap v663: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:52.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:52 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:32:52.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:52 smithi082 ceph-mon[33535]: pgmap v663: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:52.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:52 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:32:52.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:52 smithi070 ceph-mon[33250]: pgmap v663: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:52.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:52 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:32:54.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:54 smithi067 ceph-mon[29731]: pgmap v664: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:54.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:54 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:54 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:54 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:54 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:54 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:54 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:54 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:32:54.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:54 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:32:54.452 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:54 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:54 smithi082 ceph-mon[33535]: pgmap v664: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:54.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:54 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:54 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:54 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:54 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:54 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:54 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:54 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:32:54.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:54 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:32:54.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:54 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:54 smithi070 ceph-mon[33250]: pgmap v664: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:32:54.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:54 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:54 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:54 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:54 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:54 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:54 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:54 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:32:54.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:54 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:32:54.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:54 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:32:54.969 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:32:54.971+0000 7f57417fa640 1 -- 172.21.15.67:0/1530589646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f5734005eb0 con 0x7f574c1034c0 2024-05-03T17:32:54.970 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:32:54.972+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 <== mon.0 v2:172.21.15.67:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f5734005eb0 con 0x7f574c1034c0 2024-05-03T17:32:56.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:56 smithi067 ceph-mon[29731]: pgmap v665: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:56.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:56 smithi082 ceph-mon[33535]: pgmap v665: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:56.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:56 smithi070 ceph-mon[33250]: pgmap v665: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:32:58.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:32:58 smithi067 ceph-mon[29731]: pgmap v666: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:58.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:32:58 smithi082 ceph-mon[33535]: pgmap v666: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:32:58.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:32:58 smithi070 ceph-mon[33250]: pgmap v666: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:00.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:00 smithi082 ceph-mon[33535]: pgmap v667: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:00.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:00 smithi070 ceph-mon[33250]: pgmap v667: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:00.701 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:00 smithi067 ceph-mon[29731]: pgmap v667: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:01.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:01 smithi082 ceph-mon[33535]: pgmap v668: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:01.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:01 smithi070 ceph-mon[33250]: pgmap v668: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:01.701 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:01 smithi067 ceph-mon[29731]: pgmap v668: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:03.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:03 smithi067 ceph-mon[29731]: pgmap v669: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:04.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:03 smithi082 ceph-mon[33535]: pgmap v669: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:04.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:03 smithi070 ceph-mon[33250]: pgmap v669: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:05.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:05 smithi067 ceph-mon[29731]: pgmap v670: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:06.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:05 smithi082 ceph-mon[33535]: pgmap v670: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:06.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:05 smithi070 ceph-mon[33250]: pgmap v670: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:07.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:07 smithi067 ceph-mon[29731]: pgmap v671: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:08.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:07 smithi082 ceph-mon[33535]: pgmap v671: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:08.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:07 smithi070 ceph-mon[33250]: pgmap v671: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:09.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:09 smithi067 ceph-mon[29731]: pgmap v672: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:10.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:09 smithi082 ceph-mon[33535]: pgmap v672: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:10.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:09 smithi070 ceph-mon[33250]: pgmap v672: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:11.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:11 smithi067 ceph-mon[29731]: pgmap v673: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:12.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:11 smithi082 ceph-mon[33535]: pgmap v673: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:12.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:11 smithi070 ceph-mon[33250]: pgmap v673: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:13.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:13 smithi067 ceph-mon[29731]: pgmap v674: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:14.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:13 smithi082 ceph-mon[33535]: pgmap v674: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:14.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:13 smithi070 ceph-mon[33250]: pgmap v674: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:15.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:15 smithi067 ceph-mon[29731]: pgmap v675: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:16.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:15 smithi082 ceph-mon[33535]: pgmap v675: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:16.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:15 smithi070 ceph-mon[33250]: pgmap v675: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:17.951 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:17 smithi067 ceph-mon[29731]: pgmap v676: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:18.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:17 smithi082 ceph-mon[33535]: pgmap v676: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:18.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:17 smithi070 ceph-mon[33250]: pgmap v676: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:20.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:19 smithi082 ceph-mon[33535]: pgmap v677: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:20.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:19 smithi070 ceph-mon[33250]: pgmap v677: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:20.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:19 smithi067 ceph-mon[29731]: pgmap v677: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:22.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:21 smithi082 ceph-mon[33535]: pgmap v678: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:22.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:21 smithi070 ceph-mon[33250]: pgmap v678: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:22.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:21 smithi067 ceph-mon[29731]: pgmap v678: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:24.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:23 smithi082 ceph-mon[33535]: pgmap v679: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:24.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:23 smithi070 ceph-mon[33250]: pgmap v679: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:24.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:23 smithi067 ceph-mon[29731]: pgmap v679: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:26.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:25 smithi082 ceph-mon[33535]: pgmap v680: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:26.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:25 smithi070 ceph-mon[33250]: pgmap v680: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:26.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:25 smithi067 ceph-mon[29731]: pgmap v680: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:28.003 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:27 smithi067 ceph-mon[29731]: pgmap v681: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:28.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:27 smithi082 ceph-mon[33535]: pgmap v681: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:28.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:27 smithi070 ceph-mon[33250]: pgmap v681: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:30.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:29 smithi082 ceph-mon[33535]: pgmap v682: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:30.057 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:29 smithi070 ceph-mon[33250]: pgmap v682: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:30.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:29 smithi067 ceph-mon[29731]: pgmap v682: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:32.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:31 smithi082 ceph-mon[33535]: pgmap v683: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:32.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:31 smithi070 ceph-mon[33250]: pgmap v683: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:32.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:31 smithi067 ceph-mon[29731]: pgmap v683: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:34.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:33 smithi082 ceph-mon[33535]: pgmap v684: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:34.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:33 smithi070 ceph-mon[33250]: pgmap v684: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:34.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:33 smithi067 ceph-mon[29731]: pgmap v684: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:36.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:35 smithi082 ceph-mon[33535]: pgmap v685: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:36.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:35 smithi070 ceph-mon[33250]: pgmap v685: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:36.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:35 smithi067 ceph-mon[29731]: pgmap v685: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:38.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:37 smithi082 ceph-mon[33535]: pgmap v686: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:38.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:37 smithi070 ceph-mon[33250]: pgmap v686: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:38.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:37 smithi067 ceph-mon[29731]: pgmap v686: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:40.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:39 smithi082 ceph-mon[33535]: pgmap v687: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:40.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:39 smithi070 ceph-mon[33250]: pgmap v687: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:40.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:39 smithi067 ceph-mon[29731]: pgmap v687: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:42.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:41 smithi082 ceph-mon[33535]: pgmap v688: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:42.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:41 smithi070 ceph-mon[33250]: pgmap v688: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:42.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:41 smithi067 ceph-mon[29731]: pgmap v688: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:44.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:43 smithi082 ceph-mon[33535]: pgmap v689: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:44.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:43 smithi070 ceph-mon[33250]: pgmap v689: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:44.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:43 smithi067 ceph-mon[29731]: pgmap v689: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:46.013 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:45 smithi082 ceph-mon[33535]: pgmap v690: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:46.059 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:45 smithi070 ceph-mon[33250]: pgmap v690: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:46.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:45 smithi067 ceph-mon[29731]: pgmap v690: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:48.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:47 smithi082 ceph-mon[33535]: pgmap v691: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:48.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:47 smithi070 ceph-mon[33250]: pgmap v691: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:48.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:47 smithi067 ceph-mon[29731]: pgmap v691: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:50.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:49 smithi082 ceph-mon[33535]: pgmap v692: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:50.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:49 smithi070 ceph-mon[33250]: pgmap v692: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:50.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:49 smithi067 ceph-mon[29731]: pgmap v692: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:33:52.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:51 smithi082 ceph-mon[33535]: pgmap v693: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:52.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:51 smithi070 ceph-mon[33250]: pgmap v693: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:52.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:51 smithi067 ceph-mon[29731]: pgmap v693: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:54.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:53 smithi082 ceph-mon[33535]: pgmap v694: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:54.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:53 smithi070 ceph-mon[33250]: pgmap v694: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:54.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:53 smithi067 ceph-mon[29731]: pgmap v694: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:33:55.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:54 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:33:55.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:54 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:33:55.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:54 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:33:56.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:55 smithi067 ceph-mon[29731]: pgmap v695: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:56.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:33:56.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:33:56.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:55 smithi082 ceph-mon[33535]: pgmap v695: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:56.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:33:56.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:33:56.263 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:55 smithi070 ceph-mon[33250]: pgmap v695: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:33:56.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:56.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:33:56.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:33:56.307 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:33:58.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:57 smithi067 ceph-mon[29731]: pgmap v696: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:58.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:57 smithi082 ceph-mon[33535]: pgmap v696: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:33:58.306 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:57 smithi070 ceph-mon[33250]: pgmap v696: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:00.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:33:59 smithi067 ceph-mon[29731]: pgmap v697: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:00.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:33:59 smithi082 ceph-mon[33535]: pgmap v697: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:00.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:33:59 smithi070 ceph-mon[33250]: pgmap v697: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:02.107 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:01 smithi070 ceph-mon[33250]: pgmap v698: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:02.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:01 smithi067 ceph-mon[29731]: pgmap v698: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:02.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:01 smithi082 ceph-mon[33535]: pgmap v698: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:04.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:03 smithi067 ceph-mon[29731]: pgmap v699: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:04.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:03 smithi082 ceph-mon[33535]: pgmap v699: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:04.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:03 smithi070 ceph-mon[33250]: pgmap v699: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:06.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:05 smithi067 ceph-mon[29731]: pgmap v700: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:06.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:05 smithi082 ceph-mon[33535]: pgmap v700: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:06.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:05 smithi070 ceph-mon[33250]: pgmap v700: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:08.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:07 smithi067 ceph-mon[29731]: pgmap v701: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:08.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:07 smithi082 ceph-mon[33535]: pgmap v701: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:08.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:07 smithi070 ceph-mon[33250]: pgmap v701: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:10.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:09 smithi067 ceph-mon[29731]: pgmap v702: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:10.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:09 smithi082 ceph-mon[33535]: pgmap v702: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:10.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:09 smithi070 ceph-mon[33250]: pgmap v702: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:12.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:11 smithi067 ceph-mon[29731]: pgmap v703: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:12.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:11 smithi082 ceph-mon[33535]: pgmap v703: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:12.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:11 smithi070 ceph-mon[33250]: pgmap v703: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:14.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:14 smithi067 ceph-mon[29731]: pgmap v704: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:14.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:14 smithi082 ceph-mon[33535]: pgmap v704: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:14.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:14 smithi070 ceph-mon[33250]: pgmap v704: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:16.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:16 smithi067 ceph-mon[29731]: pgmap v705: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:16.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:16 smithi082 ceph-mon[33535]: pgmap v705: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:16.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:16 smithi070 ceph-mon[33250]: pgmap v705: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:18.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:18 smithi067 ceph-mon[29731]: pgmap v706: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:18.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:18 smithi082 ceph-mon[33535]: pgmap v706: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:18.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:18 smithi070 ceph-mon[33250]: pgmap v706: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:20.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:20 smithi067 ceph-mon[29731]: pgmap v707: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:20.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:20 smithi082 ceph-mon[33535]: pgmap v707: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:20.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:20 smithi070 ceph-mon[33250]: pgmap v707: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:22.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:22 smithi067 ceph-mon[29731]: pgmap v708: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:22.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:22 smithi082 ceph-mon[33535]: pgmap v708: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:22.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:22 smithi070 ceph-mon[33250]: pgmap v708: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:24.451 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:24 smithi067 ceph-mon[29731]: pgmap v709: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:24.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:24 smithi082 ceph-mon[33535]: pgmap v709: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:24.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:24 smithi070 ceph-mon[33250]: pgmap v709: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:26.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:26 smithi067 ceph-mon[29731]: pgmap v710: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:26.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:26 smithi082 ceph-mon[33535]: pgmap v710: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:26.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:26 smithi070 ceph-mon[33250]: pgmap v710: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:28.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:28 smithi067 ceph-mon[29731]: pgmap v711: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:28.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:28 smithi082 ceph-mon[33535]: pgmap v711: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:28.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:28 smithi070 ceph-mon[33250]: pgmap v711: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:30.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:30 smithi067 ceph-mon[29731]: pgmap v712: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:30.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:30 smithi082 ceph-mon[33535]: pgmap v712: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:30.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:30 smithi070 ceph-mon[33250]: pgmap v712: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:32.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:32 smithi067 ceph-mon[29731]: pgmap v713: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:32.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:32 smithi082 ceph-mon[33535]: pgmap v713: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:32.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:32 smithi070 ceph-mon[33250]: pgmap v713: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:34.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:34 smithi067 ceph-mon[29731]: pgmap v714: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:34.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:34 smithi082 ceph-mon[33535]: pgmap v714: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:34.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:34 smithi070 ceph-mon[33250]: pgmap v714: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:36.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:36 smithi067 ceph-mon[29731]: pgmap v715: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:36.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:36 smithi082 ceph-mon[33535]: pgmap v715: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:36.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:36 smithi070 ceph-mon[33250]: pgmap v715: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:38.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:38 smithi082 ceph-mon[33535]: pgmap v716: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:38.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:38 smithi070 ceph-mon[33250]: pgmap v716: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:38.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:38 smithi067 ceph-mon[29731]: pgmap v716: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:40.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:40 smithi082 ceph-mon[33535]: pgmap v717: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:40.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:40 smithi070 ceph-mon[33250]: pgmap v717: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:40.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:40 smithi067 ceph-mon[29731]: pgmap v717: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:42.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:42 smithi082 ceph-mon[33535]: pgmap v718: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:42.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:42 smithi070 ceph-mon[33250]: pgmap v718: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:42.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:42 smithi067 ceph-mon[29731]: pgmap v718: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:44.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:44 smithi082 ceph-mon[33535]: pgmap v719: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:44.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:44 smithi070 ceph-mon[33250]: pgmap v719: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:44.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:44 smithi067 ceph-mon[29731]: pgmap v719: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:46.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:46 smithi082 ceph-mon[33535]: pgmap v720: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:46.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:46 smithi070 ceph-mon[33250]: pgmap v720: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:46.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:46 smithi067 ceph-mon[29731]: pgmap v720: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:48.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:48 smithi082 ceph-mon[33535]: pgmap v721: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:48.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:48 smithi070 ceph-mon[33250]: pgmap v721: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:48.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:48 smithi067 ceph-mon[29731]: pgmap v721: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:50.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:50 smithi082 ceph-mon[33535]: pgmap v722: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:50.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:50 smithi070 ceph-mon[33250]: pgmap v722: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:50.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:50 smithi067 ceph-mon[29731]: pgmap v722: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:34:52.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:52 smithi082 ceph-mon[33535]: pgmap v723: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:52.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:52 smithi070 ceph-mon[33250]: pgmap v723: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:52.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:52 smithi067 ceph-mon[29731]: pgmap v723: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:53.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:53 smithi082 ceph-mon[33535]: pgmap v724: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:53.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:53 smithi070 ceph-mon[33250]: pgmap v724: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:53.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:53 smithi067 ceph-mon[29731]: pgmap v724: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:34:56.011 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:55 smithi082 ceph-mon[33535]: pgmap v725: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:56.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:55 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:34:56.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:55 smithi070 ceph-mon[33250]: pgmap v725: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:56.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:55 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:34:56.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:55 smithi067 ceph-mon[29731]: pgmap v725: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:34:56.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:55 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:34:57.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:56 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:34:57.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:56 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:34:57.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:56 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:34:57.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:56 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:34:57.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:56 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:34:57.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:56 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:34:57.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:56 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:34:57.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:56 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:34:57.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:56 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:34:58.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:57 smithi082 ceph-mon[33535]: pgmap v726: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:58.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:57 smithi070 ceph-mon[33250]: pgmap v726: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:34:58.101 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:57 smithi067 ceph-mon[29731]: pgmap v726: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:00.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:34:59 smithi082 ceph-mon[33535]: pgmap v727: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:00.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:34:59 smithi070 ceph-mon[33250]: pgmap v727: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:00.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:34:59 smithi067 ceph-mon[29731]: pgmap v727: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:02.014 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:01 smithi082 ceph-mon[33535]: pgmap v728: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:02.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:01 smithi070 ceph-mon[33250]: pgmap v728: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:02.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:01 smithi067 ceph-mon[29731]: pgmap v728: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:04.011 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:03 smithi082 ceph-mon[33535]: pgmap v729: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:04.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:03 smithi070 ceph-mon[33250]: pgmap v729: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:04.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:03 smithi067 ceph-mon[29731]: pgmap v729: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:06.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:05 smithi082 ceph-mon[33535]: pgmap v730: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:06.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:05 smithi070 ceph-mon[33250]: pgmap v730: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:06.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:05 smithi067 ceph-mon[29731]: pgmap v730: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:08.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:07 smithi082 ceph-mon[33535]: pgmap v731: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:08.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:07 smithi070 ceph-mon[33250]: pgmap v731: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:08.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:07 smithi067 ceph-mon[29731]: pgmap v731: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:10.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:09 smithi082 ceph-mon[33535]: pgmap v732: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:10.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:09 smithi070 ceph-mon[33250]: pgmap v732: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:10.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:09 smithi067 ceph-mon[29731]: pgmap v732: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:12.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:11 smithi082 ceph-mon[33535]: pgmap v733: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:12.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:11 smithi070 ceph-mon[33250]: pgmap v733: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:12.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:11 smithi067 ceph-mon[29731]: pgmap v733: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:14.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:13 smithi082 ceph-mon[33535]: pgmap v734: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:14.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:13 smithi070 ceph-mon[33250]: pgmap v734: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:14.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:13 smithi067 ceph-mon[29731]: pgmap v734: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:16.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:15 smithi082 ceph-mon[33535]: pgmap v735: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:16.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:15 smithi070 ceph-mon[33250]: pgmap v735: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:16.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:15 smithi067 ceph-mon[29731]: pgmap v735: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:18.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:17 smithi082 ceph-mon[33535]: pgmap v736: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:18.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:17 smithi070 ceph-mon[33250]: pgmap v736: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:18.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:17 smithi067 ceph-mon[29731]: pgmap v736: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:20.011 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:19 smithi082 ceph-mon[33535]: pgmap v737: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:20.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:19 smithi070 ceph-mon[33250]: pgmap v737: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:20.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:19 smithi067 ceph-mon[29731]: pgmap v737: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:22.012 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:21 smithi082 ceph-mon[33535]: pgmap v738: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:22.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:21 smithi070 ceph-mon[33250]: pgmap v738: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:22.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:21 smithi067 ceph-mon[29731]: pgmap v738: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:24.011 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:23 smithi082 ceph-mon[33535]: pgmap v739: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:24.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:23 smithi070 ceph-mon[33250]: pgmap v739: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:24.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:23 smithi067 ceph-mon[29731]: pgmap v739: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:26.011 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:25 smithi082 ceph-mon[33535]: pgmap v740: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:26.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:25 smithi070 ceph-mon[33250]: pgmap v740: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:26.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:25 smithi067 ceph-mon[29731]: pgmap v740: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:28.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:27 smithi070 ceph-mon[33250]: pgmap v741: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:28.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:27 smithi067 ceph-mon[29731]: pgmap v741: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:28.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:27 smithi082 ceph-mon[33535]: pgmap v741: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:30.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:29 smithi070 ceph-mon[33250]: pgmap v742: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:30.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:29 smithi067 ceph-mon[29731]: pgmap v742: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:30.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:29 smithi082 ceph-mon[33535]: pgmap v742: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:32.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:31 smithi070 ceph-mon[33250]: pgmap v743: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:35:32.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:31 smithi067 ceph-mon[29731]: pgmap v743: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:35:32.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:31 smithi082 ceph-mon[33535]: pgmap v743: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:35:34.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:33 smithi070 ceph-mon[33250]: pgmap v744: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:35:34.064 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:33 smithi082 ceph-mon[33535]: pgmap v744: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:35:34.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:33 smithi067 ceph-mon[29731]: pgmap v744: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:35:36.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:35 smithi070 ceph-mon[33250]: pgmap v745: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:35:36.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:35 smithi067 ceph-mon[29731]: pgmap v745: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:35:36.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:35 smithi082 ceph-mon[33535]: pgmap v745: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:35:38.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:37 smithi070 ceph-mon[33250]: pgmap v746: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T17:35:38.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:37 smithi067 ceph-mon[29731]: pgmap v746: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T17:35:38.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:37 smithi082 ceph-mon[33535]: pgmap v746: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-05-03T17:35:40.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:39 smithi070 ceph-mon[33250]: pgmap v747: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:35:40.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:39 smithi067 ceph-mon[29731]: pgmap v747: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:35:40.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:39 smithi082 ceph-mon[33535]: pgmap v747: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-05-03T17:35:42.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:41 smithi070 ceph-mon[33250]: pgmap v748: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:35:42.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:41 smithi067 ceph-mon[29731]: pgmap v748: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:35:42.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:41 smithi082 ceph-mon[33535]: pgmap v748: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-05-03T17:35:44.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:43 smithi070 ceph-mon[33250]: pgmap v749: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:44.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:43 smithi067 ceph-mon[29731]: pgmap v749: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:44.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:43 smithi082 ceph-mon[33535]: pgmap v749: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:46.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:45 smithi070 ceph-mon[33250]: pgmap v750: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:46.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:45 smithi067 ceph-mon[29731]: pgmap v750: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:46.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:45 smithi082 ceph-mon[33535]: pgmap v750: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:48.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:47 smithi070 ceph-mon[33250]: pgmap v751: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:48.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:47 smithi067 ceph-mon[29731]: pgmap v751: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:48.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:47 smithi082 ceph-mon[33535]: pgmap v751: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:50.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:49 smithi070 ceph-mon[33250]: pgmap v752: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:50.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:49 smithi067 ceph-mon[29731]: pgmap v752: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:50.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:49 smithi082 ceph-mon[33535]: pgmap v752: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:35:52.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:51 smithi070 ceph-mon[33250]: pgmap v753: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:52.201 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:51 smithi067 ceph-mon[29731]: pgmap v753: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:52.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:51 smithi082 ceph-mon[33535]: pgmap v753: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:54.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:53 smithi070 ceph-mon[33250]: pgmap v754: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:54.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:53 smithi067 ceph-mon[29731]: pgmap v754: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:54.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:53 smithi082 ceph-mon[33535]: pgmap v754: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:35:54.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:35:54.997+0000 7f57417fa640 1 -- 172.21.15.67:0/1530589646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f57340060f0 con 0x7f574c1034c0 2024-05-03T17:35:54.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:35:54.997+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 <== mon.0 v2:172.21.15.67:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f57340060f0 con 0x7f574c1034c0 2024-05-03T17:35:56.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:55 smithi067 ceph-mon[29731]: pgmap v755: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:56.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:55 smithi082 ceph-mon[33535]: pgmap v755: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:56.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:55 smithi070 ceph-mon[33250]: pgmap v755: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:35:57.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:56 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:35:57.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:56 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:35:57.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:56 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:35:58.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:57 smithi067 ceph-mon[29731]: pgmap v756: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:58.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:57 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:35:58.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:57 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:35:58.102 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:57 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:35:58.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:57 smithi082 ceph-mon[33535]: pgmap v756: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:58.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:57 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:35:58.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:57 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:35:58.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:57 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:35:58.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:57 smithi070 ceph-mon[33250]: pgmap v756: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:35:58.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:57 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:35:58.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:57 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:35:58.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:57 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:36:00.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:35:59 smithi067 ceph-mon[29731]: pgmap v757: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:36:00.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:35:59 smithi082 ceph-mon[33535]: pgmap v757: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:36:00.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:35:59 smithi070 ceph-mon[33250]: pgmap v757: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:36:02.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:01 smithi067 ceph-mon[29731]: pgmap v758: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:02.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:01 smithi082 ceph-mon[33535]: pgmap v758: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:02.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:01 smithi070 ceph-mon[33250]: pgmap v758: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:04.107 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:03 smithi070 ceph-mon[33250]: pgmap v759: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:04.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:03 smithi067 ceph-mon[29731]: pgmap v759: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:04.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:03 smithi082 ceph-mon[33535]: pgmap v759: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:06.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:05 smithi067 ceph-mon[29731]: pgmap v760: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:06.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:05 smithi082 ceph-mon[33535]: pgmap v760: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:06.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:05 smithi070 ceph-mon[33250]: pgmap v760: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:08.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:07 smithi067 ceph-mon[29731]: pgmap v761: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:08.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:07 smithi082 ceph-mon[33535]: pgmap v761: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:08.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:07 smithi070 ceph-mon[33250]: pgmap v761: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:10.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:09 smithi067 ceph-mon[29731]: pgmap v762: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:36:10.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:09 smithi082 ceph-mon[33535]: pgmap v762: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:36:10.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:09 smithi070 ceph-mon[33250]: pgmap v762: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:36:12.202 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:11 smithi067 ceph-mon[29731]: pgmap v763: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:12.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:11 smithi082 ceph-mon[33535]: pgmap v763: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:12.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:11 smithi070 ceph-mon[33250]: pgmap v763: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:14.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:14 smithi067 ceph-mon[29731]: pgmap v764: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:14.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:14 smithi082 ceph-mon[33535]: pgmap v764: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:14.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:14 smithi070 ceph-mon[33250]: pgmap v764: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:16.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:16 smithi067 ceph-mon[29731]: pgmap v765: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:16.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:16 smithi082 ceph-mon[33535]: pgmap v765: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:16.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:16 smithi070 ceph-mon[33250]: pgmap v765: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:18.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:18 smithi067 ceph-mon[29731]: pgmap v766: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:18.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:18 smithi082 ceph-mon[33535]: pgmap v766: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:18.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:18 smithi070 ceph-mon[33250]: pgmap v766: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:20.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:20 smithi082 ceph-mon[33535]: pgmap v767: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:36:20.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:20 smithi070 ceph-mon[33250]: pgmap v767: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:36:20.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:20 smithi067 ceph-mon[29731]: pgmap v767: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:36:22.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:22 smithi082 ceph-mon[33535]: pgmap v768: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:22.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:22 smithi070 ceph-mon[33250]: pgmap v768: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:22.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:22 smithi067 ceph-mon[29731]: pgmap v768: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:24.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:24 smithi082 ceph-mon[33535]: pgmap v769: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:24.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:24 smithi070 ceph-mon[33250]: pgmap v769: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:24.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:24 smithi067 ceph-mon[29731]: pgmap v769: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:26.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:26 smithi082 ceph-mon[33535]: pgmap v770: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:26.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:26 smithi070 ceph-mon[33250]: pgmap v770: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:26.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:26 smithi067 ceph-mon[29731]: pgmap v770: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:28.502 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:28 smithi067 ceph-mon[29731]: pgmap v771: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:28.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:28 smithi082 ceph-mon[33535]: pgmap v771: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:28.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:28 smithi070 ceph-mon[33250]: pgmap v771: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:30.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:30 smithi082 ceph-mon[33535]: pgmap v772: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:30.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:30 smithi070 ceph-mon[33250]: pgmap v772: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:30.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:30 smithi067 ceph-mon[29731]: pgmap v772: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:32.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:32 smithi082 ceph-mon[33535]: pgmap v773: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:32.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:32 smithi070 ceph-mon[33250]: pgmap v773: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:32.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:32 smithi067 ceph-mon[29731]: pgmap v773: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:34.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:34 smithi082 ceph-mon[33535]: pgmap v774: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:34.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:34 smithi070 ceph-mon[33250]: pgmap v774: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:34.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:34 smithi067 ceph-mon[29731]: pgmap v774: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:36.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:36 smithi082 ceph-mon[33535]: pgmap v775: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:36.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:36 smithi070 ceph-mon[33250]: pgmap v775: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:36.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:36 smithi067 ceph-mon[29731]: pgmap v775: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:38.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:38 smithi082 ceph-mon[33535]: pgmap v776: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:38.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:38 smithi070 ceph-mon[33250]: pgmap v776: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:38.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:38 smithi067 ceph-mon[29731]: pgmap v776: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:40.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:40 smithi082 ceph-mon[33535]: pgmap v777: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:40.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:40 smithi070 ceph-mon[33250]: pgmap v777: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:40.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:40 smithi067 ceph-mon[29731]: pgmap v777: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:42.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:42 smithi082 ceph-mon[33535]: pgmap v778: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:42.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:42 smithi070 ceph-mon[33250]: pgmap v778: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:42.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:42 smithi067 ceph-mon[29731]: pgmap v778: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:44.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:44 smithi082 ceph-mon[33535]: pgmap v779: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:44.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:44 smithi070 ceph-mon[33250]: pgmap v779: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:44.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:44 smithi067 ceph-mon[29731]: pgmap v779: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:45.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:45 smithi070 ceph-mon[33250]: pgmap v780: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:45.700 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:45 smithi067 ceph-mon[29731]: pgmap v780: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:45.761 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:45 smithi082 ceph-mon[33535]: pgmap v780: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:48.011 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:47 smithi082 ceph-mon[33535]: pgmap v781: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:48.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:47 smithi070 ceph-mon[33250]: pgmap v781: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:48.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:47 smithi067 ceph-mon[29731]: pgmap v781: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:50.011 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:49 smithi082 ceph-mon[33535]: pgmap v782: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:36:50.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:49 smithi070 ceph-mon[33250]: pgmap v782: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:36:50.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:49 smithi067 ceph-mon[29731]: pgmap v782: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:36:52.011 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:51 smithi082 ceph-mon[33535]: pgmap v783: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:52.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:51 smithi070 ceph-mon[33250]: pgmap v783: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:52.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:51 smithi067 ceph-mon[29731]: pgmap v783: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:54.011 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:53 smithi082 ceph-mon[33535]: pgmap v784: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:54.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:53 smithi070 ceph-mon[33250]: pgmap v784: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:54.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:53 smithi067 ceph-mon[29731]: pgmap v784: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:36:56.011 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:55 smithi082 ceph-mon[33535]: pgmap v785: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:56.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:55 smithi070 ceph-mon[33250]: pgmap v785: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:56.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:55 smithi067 ceph-mon[29731]: pgmap v785: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:36:58.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:57 smithi070 ceph-mon[33250]: pgmap v786: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:58.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:57 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:36:58.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:57 smithi067 ceph-mon[29731]: pgmap v786: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:58.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:57 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:36:58.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:57 smithi082 ceph-mon[33535]: pgmap v786: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:36:58.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:57 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:36:59.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:58 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:36:59.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:58 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:36:59.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:58 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:36:59.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:36:59.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:36:59.200 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:58 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:36:59.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:36:59.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:36:59.262 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:58 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:37:00.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:36:59 smithi082 ceph-mon[33535]: pgmap v787: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:00.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:36:59 smithi070 ceph-mon[33250]: pgmap v787: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:00.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:36:59 smithi067 ceph-mon[29731]: pgmap v787: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:02.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:02 smithi082 ceph-mon[33535]: pgmap v788: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:02.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:02 smithi070 ceph-mon[33250]: pgmap v788: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:02.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:02 smithi067 ceph-mon[29731]: pgmap v788: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:04.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:04 smithi082 ceph-mon[33535]: pgmap v789: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:04.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:04 smithi070 ceph-mon[33250]: pgmap v789: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:04.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:04 smithi067 ceph-mon[29731]: pgmap v789: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:06.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:06 smithi082 ceph-mon[33535]: pgmap v790: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:06.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:06 smithi070 ceph-mon[33250]: pgmap v790: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:06.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:06 smithi067 ceph-mon[29731]: pgmap v790: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:08.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:08 smithi082 ceph-mon[33535]: pgmap v791: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:08.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:08 smithi070 ceph-mon[33250]: pgmap v791: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:08.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:08 smithi067 ceph-mon[29731]: pgmap v791: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:10.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:10 smithi082 ceph-mon[33535]: pgmap v792: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:10.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:10 smithi070 ceph-mon[33250]: pgmap v792: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:10.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:10 smithi067 ceph-mon[29731]: pgmap v792: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:12.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:12 smithi070 ceph-mon[33250]: pgmap v793: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:12.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:12 smithi067 ceph-mon[29731]: pgmap v793: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:12.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:12 smithi082 ceph-mon[33535]: pgmap v793: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:14.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:14 smithi070 ceph-mon[33250]: pgmap v794: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:14.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:14 smithi067 ceph-mon[29731]: pgmap v794: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:14.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:14 smithi082 ceph-mon[33535]: pgmap v794: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:16.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:16 smithi070 ceph-mon[33250]: pgmap v795: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:16.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:16 smithi067 ceph-mon[29731]: pgmap v795: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:16.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:16 smithi082 ceph-mon[33535]: pgmap v795: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:18.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:18 smithi070 ceph-mon[33250]: pgmap v796: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:18.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:18 smithi067 ceph-mon[29731]: pgmap v796: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:18.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:18 smithi082 ceph-mon[33535]: pgmap v796: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:20.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:20 smithi070 ceph-mon[33250]: pgmap v797: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:20.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:20 smithi067 ceph-mon[29731]: pgmap v797: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:20.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:20 smithi082 ceph-mon[33535]: pgmap v797: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:22.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:22 smithi070 ceph-mon[33250]: pgmap v798: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:22.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:22 smithi067 ceph-mon[29731]: pgmap v798: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:22.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:22 smithi082 ceph-mon[33535]: pgmap v798: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:24.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:24 smithi070 ceph-mon[33250]: pgmap v799: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:24.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:24 smithi067 ceph-mon[29731]: pgmap v799: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:24.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:24 smithi082 ceph-mon[33535]: pgmap v799: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:26.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:26 smithi070 ceph-mon[33250]: pgmap v800: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:26.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:26 smithi067 ceph-mon[29731]: pgmap v800: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:26.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:26 smithi082 ceph-mon[33535]: pgmap v800: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:28.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:28 smithi070 ceph-mon[33250]: pgmap v801: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:28.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:28 smithi067 ceph-mon[29731]: pgmap v801: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:28.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:28 smithi082 ceph-mon[33535]: pgmap v801: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:30.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:30 smithi070 ceph-mon[33250]: pgmap v802: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:30.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:30 smithi067 ceph-mon[29731]: pgmap v802: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:30.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:30 smithi082 ceph-mon[33535]: pgmap v802: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:32.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:32 smithi070 ceph-mon[33250]: pgmap v803: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:32.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:32 smithi067 ceph-mon[29731]: pgmap v803: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:32.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:32 smithi082 ceph-mon[33535]: pgmap v803: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:34.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:34 smithi070 ceph-mon[33250]: pgmap v804: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:34.313 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:34 smithi082 ceph-mon[33535]: pgmap v804: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:34.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:34 smithi067 ceph-mon[29731]: pgmap v804: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:36.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:36 smithi067 ceph-mon[29731]: pgmap v805: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:36.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:36 smithi082 ceph-mon[33535]: pgmap v805: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:36.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:36 smithi070 ceph-mon[33250]: pgmap v805: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:38.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:38 smithi067 ceph-mon[29731]: pgmap v806: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:38.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:38 smithi082 ceph-mon[33535]: pgmap v806: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:38.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:38 smithi070 ceph-mon[33250]: pgmap v806: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:40.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:40 smithi067 ceph-mon[29731]: pgmap v807: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:40.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:40 smithi082 ceph-mon[33535]: pgmap v807: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:40.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:40 smithi070 ceph-mon[33250]: pgmap v807: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:42.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:42 smithi067 ceph-mon[29731]: pgmap v808: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:42.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:42 smithi082 ceph-mon[33535]: pgmap v808: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:42.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:42 smithi070 ceph-mon[33250]: pgmap v808: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:44.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:44 smithi067 ceph-mon[29731]: pgmap v809: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:44.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:44 smithi082 ceph-mon[33535]: pgmap v809: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:44.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:44 smithi070 ceph-mon[33250]: pgmap v809: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:46.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:46 smithi067 ceph-mon[29731]: pgmap v810: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:46.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:46 smithi082 ceph-mon[33535]: pgmap v810: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:46.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:46 smithi070 ceph-mon[33250]: pgmap v810: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:48.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:48 smithi067 ceph-mon[29731]: pgmap v811: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:48.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:48 smithi082 ceph-mon[33535]: pgmap v811: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:48.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:48 smithi070 ceph-mon[33250]: pgmap v811: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:50.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:50 smithi067 ceph-mon[29731]: pgmap v812: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:50.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:50 smithi082 ceph-mon[33535]: pgmap v812: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:50.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:50 smithi070 ceph-mon[33250]: pgmap v812: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:37:52.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:52 smithi067 ceph-mon[29731]: pgmap v813: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:52.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:52 smithi082 ceph-mon[33535]: pgmap v813: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:52.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:52 smithi070 ceph-mon[33250]: pgmap v813: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:54.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:54 smithi067 ceph-mon[29731]: pgmap v814: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:54.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:54 smithi082 ceph-mon[33535]: pgmap v814: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:54.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:54 smithi070 ceph-mon[33250]: pgmap v814: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:37:56.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:56 smithi067 ceph-mon[29731]: pgmap v815: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:56.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:56 smithi082 ceph-mon[33535]: pgmap v815: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:56.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:56 smithi070 ceph-mon[33250]: pgmap v815: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:37:58.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:58 smithi067 ceph-mon[29731]: pgmap v816: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:58.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:58 smithi082 ceph-mon[33535]: pgmap v816: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:58.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:58 smithi070 ceph-mon[33250]: pgmap v816: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:37:59.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:37:59.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:37:59.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:37:59.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:37:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:37:59.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:37:59.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:37:59.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:37:59.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:37:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:37:59.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:37:59.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:37:59.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:37:59.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:37:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:38:00.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:00 smithi067 ceph-mon[29731]: pgmap v817: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:00.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:00 smithi082 ceph-mon[33535]: pgmap v817: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:00.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:00 smithi070 ceph-mon[33250]: pgmap v817: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:02.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:02 smithi067 ceph-mon[29731]: pgmap v818: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:02.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:02 smithi082 ceph-mon[33535]: pgmap v818: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:02.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:02 smithi070 ceph-mon[33250]: pgmap v818: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:04.357 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:04 smithi070 ceph-mon[33250]: pgmap v819: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:04.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:04 smithi067 ceph-mon[29731]: pgmap v819: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:04.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:04 smithi082 ceph-mon[33535]: pgmap v819: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:06.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:06 smithi067 ceph-mon[29731]: pgmap v820: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:06.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:06 smithi082 ceph-mon[33535]: pgmap v820: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:06.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:06 smithi070 ceph-mon[33250]: pgmap v820: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:08.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:08 smithi067 ceph-mon[29731]: pgmap v821: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:08.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:08 smithi082 ceph-mon[33535]: pgmap v821: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:08.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:08 smithi070 ceph-mon[33250]: pgmap v821: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:10.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:10 smithi067 ceph-mon[29731]: pgmap v822: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:10.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:10 smithi082 ceph-mon[33535]: pgmap v822: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:10.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:10 smithi070 ceph-mon[33250]: pgmap v822: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:12.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:12 smithi067 ceph-mon[29731]: pgmap v823: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:12.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:12 smithi082 ceph-mon[33535]: pgmap v823: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:12.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:12 smithi070 ceph-mon[33250]: pgmap v823: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:14.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:14 smithi067 ceph-mon[29731]: pgmap v824: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:14.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:14 smithi082 ceph-mon[33535]: pgmap v824: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:14.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:14 smithi070 ceph-mon[33250]: pgmap v824: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:16.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:16 smithi067 ceph-mon[29731]: pgmap v825: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:16.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:16 smithi082 ceph-mon[33535]: pgmap v825: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:16.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:16 smithi070 ceph-mon[33250]: pgmap v825: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:18.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:18 smithi067 ceph-mon[29731]: pgmap v826: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:18.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:18 smithi082 ceph-mon[33535]: pgmap v826: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:18.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:18 smithi070 ceph-mon[33250]: pgmap v826: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:20.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:20 smithi067 ceph-mon[29731]: pgmap v827: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:20.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:20 smithi082 ceph-mon[33535]: pgmap v827: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:20.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:20 smithi070 ceph-mon[33250]: pgmap v827: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:22.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:22 smithi067 ceph-mon[29731]: pgmap v828: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:22.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:22 smithi082 ceph-mon[33535]: pgmap v828: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:22.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:22 smithi070 ceph-mon[33250]: pgmap v828: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:24.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:24 smithi067 ceph-mon[29731]: pgmap v829: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:24.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:24 smithi082 ceph-mon[33535]: pgmap v829: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:24.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:24 smithi070 ceph-mon[33250]: pgmap v829: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:26.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:26 smithi067 ceph-mon[29731]: pgmap v830: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:26.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:26 smithi082 ceph-mon[33535]: pgmap v830: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:26.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:26 smithi070 ceph-mon[33250]: pgmap v830: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:28.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:28 smithi067 ceph-mon[29731]: pgmap v831: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:28.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:28 smithi082 ceph-mon[33535]: pgmap v831: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:28.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:28 smithi070 ceph-mon[33250]: pgmap v831: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:30.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:30 smithi067 ceph-mon[29731]: pgmap v832: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:30.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:30 smithi082 ceph-mon[33535]: pgmap v832: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:30.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:30 smithi070 ceph-mon[33250]: pgmap v832: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:32.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:32 smithi067 ceph-mon[29731]: pgmap v833: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:32.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:32 smithi082 ceph-mon[33535]: pgmap v833: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:32.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:32 smithi070 ceph-mon[33250]: pgmap v833: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:34.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:34 smithi067 ceph-mon[29731]: pgmap v834: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:34.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:34 smithi082 ceph-mon[33535]: pgmap v834: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:34.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:34 smithi070 ceph-mon[33250]: pgmap v834: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:36.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:36 smithi067 ceph-mon[29731]: pgmap v835: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:36.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:36 smithi082 ceph-mon[33535]: pgmap v835: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:36.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:36 smithi070 ceph-mon[33250]: pgmap v835: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:38.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:38 smithi082 ceph-mon[33535]: pgmap v836: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:38.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:38 smithi070 ceph-mon[33250]: pgmap v836: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:38.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:38 smithi067 ceph-mon[29731]: pgmap v836: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:40.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:40 smithi082 ceph-mon[33535]: pgmap v837: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:40.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:40 smithi070 ceph-mon[33250]: pgmap v837: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:40.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:40 smithi067 ceph-mon[29731]: pgmap v837: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:42.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:42 smithi082 ceph-mon[33535]: pgmap v838: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:42.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:42 smithi070 ceph-mon[33250]: pgmap v838: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:42.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:42 smithi067 ceph-mon[29731]: pgmap v838: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:44.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:44 smithi082 ceph-mon[33535]: pgmap v839: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:44.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:44 smithi070 ceph-mon[33250]: pgmap v839: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:44.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:44 smithi067 ceph-mon[29731]: pgmap v839: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:46.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:46 smithi082 ceph-mon[33535]: pgmap v840: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:46.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:46 smithi070 ceph-mon[33250]: pgmap v840: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:46.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:46 smithi067 ceph-mon[29731]: pgmap v840: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:48.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:48 smithi082 ceph-mon[33535]: pgmap v841: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:48.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:48 smithi070 ceph-mon[33250]: pgmap v841: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:48.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:48 smithi067 ceph-mon[29731]: pgmap v841: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:50.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:50 smithi082 ceph-mon[33535]: pgmap v842: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:50.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:50 smithi070 ceph-mon[33250]: pgmap v842: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:50.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:50 smithi067 ceph-mon[29731]: pgmap v842: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:38:52.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:52 smithi082 ceph-mon[33535]: pgmap v843: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:52.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:52 smithi070 ceph-mon[33250]: pgmap v843: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:52.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:52 smithi067 ceph-mon[29731]: pgmap v843: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:54.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:54 smithi082 ceph-mon[33535]: pgmap v844: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:54.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:54 smithi070 ceph-mon[33250]: pgmap v844: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:54.699 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:54 smithi067 ceph-mon[29731]: pgmap v844: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-03T17:38:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:54.911+0000 7f5750c75640 1 -- 172.21.15.67:0/1530589646 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f57380b2870 msgr2=0x7f57380b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900012066) for more than 900000000 us, fault. 2024-05-03T17:38:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:54.911+0000 7f5750c75640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f57380b2870 0x7f57380b4d30 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f573c006fd0 tx=0x7f573c008040 comp rx=0 tx=0).stop 2024-05-03T17:38:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:54.911+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f57380b2870 msgr2=0x7f57380b4d30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-03T17:38:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:54.911+0000 7f57427fc640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f57380b2870 0x7f57380b4d30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:38:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:54.911+0000 7f57427fc640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f57380bc900 0x7f57380becf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:38:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:54.911+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 --> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f57380bf2d0 con 0x7f57380bc900 2024-05-03T17:38:54.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:54.911+0000 7f5750c75640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f57380bc900 0x7f57380becf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:38:54.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:54.912+0000 7f5750c75640 1 --2- 172.21.15.67:0/1530589646 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f57380bc900 0x7f57380becf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f573c00a1a0 tx=0x7f573c011000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:38:55.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:55.022+0000 7f57417fa640 1 -- 172.21.15.67:0/1530589646 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f57340063b0 con 0x7f574c1034c0 2024-05-03T17:38:55.020 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:55.023+0000 7f57427fc640 1 -- 172.21.15.67:0/1530589646 <== mon.0 v2:172.21.15.67:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f57480b8100 con 0x7f574c1034c0 2024-05-03T17:38:55.120 INFO:teuthology.orchestra.run.smithi067.stderr:Error ETIMEDOUT: Connection timed out. Please check the client's permission and connection. 2024-05-03T17:38:55.127 DEBUG:teuthology.orchestra.run:got remote process result: 110 2024-05-03T17:38:55.128 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-05-03T17:38:55.371 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f0bf7852e935487296eb382aca021d20 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10'" 2024-05-03T17:38:55.373 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-03T17:38:55.387 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-03T17:38:55.387 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:38:55.387 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-03T17:38:55.411 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2024-05-03T17:38:55.411 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-03T17:38:55.471 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd blocklist ls 2024-05-03T17:38:55.768 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:38:56.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:56 smithi067 ceph-mon[29731]: pgmap v845: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:56.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:56 smithi067 ceph-mon[29731]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:38:56.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:56 smithi082 ceph-mon[33535]: pgmap v845: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:56.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:56 smithi082 ceph-mon[33535]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:38:56.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:56 smithi070 ceph-mon[33250]: pgmap v845: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-03T17:38:56.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:56 smithi070 ceph-mon[33250]: from='client.15105 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-05-03T17:38:57.663 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.664+0000 7fc6daadc700 1 -- 172.21.15.67:0/1903343969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d4108870 msgr2=0x7fc6d41103f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:38:57.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.664+0000 7fc6daadc700 1 --2- 172.21.15.67:0/1903343969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d4108870 0x7fc6d41103f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc00b0d0 tx=0x7fc6cc00b3e0 comp rx=0 tx=0).stop 2024-05-03T17:38:57.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.664+0000 7fc6daadc700 1 -- 172.21.15.67:0/1903343969 shutdown_connections 2024-05-03T17:38:57.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.664+0000 7fc6daadc700 1 --2- 172.21.15.67:0/1903343969 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d4108870 0x7fc6d41103f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:38:57.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.664+0000 7fc6daadc700 1 --2- 172.21.15.67:0/1903343969 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6d4103780 0x7fc6d4103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:38:57.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.664+0000 7fc6daadc700 1 --2- 172.21.15.67:0/1903343969 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6d4104130 0x7fc6d4108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:38:57.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.664+0000 7fc6daadc700 1 -- 172.21.15.67:0/1903343969 >> 172.21.15.67:0/1903343969 conn(0x7fc6d407a610 msgr2=0x7fc6d407aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:38:57.664 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.664+0000 7fc6daadc700 1 -- 172.21.15.67:0/1903343969 shutdown_connections 2024-05-03T17:38:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.664+0000 7fc6daadc700 1 -- 172.21.15.67:0/1903343969 wait complete. 2024-05-03T17:38:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.664+0000 7fc6daadc700 1 Processor -- start 2024-05-03T17:38:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.664+0000 7fc6daadc700 1 -- start start 2024-05-03T17:38:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6daadc700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6d4103780 0x7fc6d41a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:38:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6daadc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d4104130 0x7fc6d41a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:38:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6daadc700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6d4108870 0x7fc6d419bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:38:57.665 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6daadc700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d4113660 con 0x7fc6d4104130 2024-05-03T17:38:57.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6daadc700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d41134e0 con 0x7fc6d4108870 2024-05-03T17:38:57.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6daadc700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d41137e0 con 0x7fc6d4103780 2024-05-03T17:38:57.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6d3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d4104130 0x7fc6d41a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:38:57.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6d3fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d4104130 0x7fc6d41a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57544/0 (socket says 172.21.15.67:57544) 2024-05-03T17:38:57.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6d3fff700 1 -- 172.21.15.67:0/833578242 learned_addr learned my addr 172.21.15.67:0/833578242 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:38:57.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6d8878700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6d4103780 0x7fc6d41a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:38:57.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6d9079700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6d4108870 0x7fc6d419bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:38:57.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6d3fff700 1 -- 172.21.15.67:0/833578242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6d4103780 msgr2=0x7fc6d41a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:38:57.666 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6d3fff700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6d4103780 0x7fc6d41a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:38:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6d3fff700 1 -- 172.21.15.67:0/833578242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6d4108870 msgr2=0x7fc6d419bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:38:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6d3fff700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6d4108870 0x7fc6d419bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:38:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6d3fff700 1 -- 172.21.15.67:0/833578242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d419c220 con 0x7fc6d4104130 2024-05-03T17:38:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6d3fff700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d4104130 0x7fc6d41a1e40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000d980 tx=0x7fc6c000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:38:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.665+0000 7fc6d1ffb700 1 -- 172.21.15.67:0/833578242 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0017070 con 0x7fc6d4104130 2024-05-03T17:38:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.666+0000 7fc6daadc700 1 -- 172.21.15.67:0/833578242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d419c510 con 0x7fc6d4104130 2024-05-03T17:38:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.666+0000 7fc6d1ffb700 1 -- 172.21.15.67:0/833578242 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6c0005020 con 0x7fc6d4104130 2024-05-03T17:38:57.667 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.666+0000 7fc6d1ffb700 1 -- 172.21.15.67:0/833578242 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c000fa10 con 0x7fc6d4104130 2024-05-03T17:38:57.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.666+0000 7fc6daadc700 1 -- 172.21.15.67:0/833578242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d41a87d0 con 0x7fc6d4104130 2024-05-03T17:38:57.668 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.667+0000 7fc6daadc700 1 -- 172.21.15.67:0/833578242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d406a190 con 0x7fc6d4104130 2024-05-03T17:38:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.668+0000 7fc6d1ffb700 1 -- 172.21.15.67:0/833578242 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fc6c0004a30 con 0x7fc6d4104130 2024-05-03T17:38:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.669+0000 7fc6d1ffb700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fc6c40b2910 0x7fc6c40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:38:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.669+0000 7fc6d8878700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fc6c40b2910 0x7fc6c40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:38:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.669+0000 7fc6d1ffb700 1 -- 172.21.15.67:0/833578242 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fc6c00e2680 con 0x7fc6d4104130 2024-05-03T17:38:57.669 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.669+0000 7fc6d8878700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fc6c40b2910 0x7fc6c40b4dd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc6d419cc00 tx=0x7fc6c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:38:57.670 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.672+0000 7fc6d1ffb700 1 -- 172.21.15.67:0/833578242 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fc6c0020020 con 0x7fc6d4104130 2024-05-03T17:38:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.909+0000 7fc6daadc700 1 -- 172.21.15.67:0/833578242 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc6d41a8a00 con 0x7fc6d4104130 2024-05-03T17:38:57.907 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.909+0000 7fc6d1ffb700 1 -- 172.21.15.67:0/833578242 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v81) ==== 81+0+591 (secure 0 0 0) 0x7fc6c00aa230 con 0x7fc6d4104130 2024-05-03T17:38:57.907 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2131289004 2024-05-04T17:13:06.324002+0000 2024-05-03T17:38:57.907 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/3335149382 2024-05-04T17:13:06.324002+0000 2024-05-03T17:38:57.908 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/992151181 2024-05-04T17:13:06.324002+0000 2024-05-03T17:38:57.908 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6801/1345903711 2024-05-04T17:12:38.899429+0000 2024-05-03T17:38:57.908 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/1345903711 2024-05-04T17:12:38.899429+0000 2024-05-03T17:38:57.908 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2614686067 2024-05-04T17:12:38.899429+0000 2024-05-03T17:38:57.908 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/3137329076 2024-05-04T17:12:38.899429+0000 2024-05-03T17:38:57.908 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2323686292 2024-05-04T17:13:06.324002+0000 2024-05-03T17:38:57.908 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:0/2579128777 2024-05-04T17:12:38.899429+0000 2024-05-03T17:38:57.909 INFO:teuthology.orchestra.run.smithi067.stdout:172.21.15.67:6800/3335149382 2024-05-04T17:13:06.324002+0000 2024-05-03T17:38:57.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.911+0000 7fc6daadc700 1 -- 172.21.15.67:0/833578242 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fc6c40b2910 msgr2=0x7fc6c40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:38:57.909 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.911+0000 7fc6daadc700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fc6c40b2910 0x7fc6c40b4dd0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fc6d419cc00 tx=0x7fc6c8009260 comp rx=0 tx=0).stop 2024-05-03T17:38:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.911+0000 7fc6daadc700 1 -- 172.21.15.67:0/833578242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d4104130 msgr2=0x7fc6d41a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:38:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.911+0000 7fc6daadc700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d4104130 0x7fc6d41a1e40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000d980 tx=0x7fc6c000dc90 comp rx=0 tx=0).stop 2024-05-03T17:38:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.911+0000 7fc6daadc700 1 -- 172.21.15.67:0/833578242 shutdown_connections 2024-05-03T17:38:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.911+0000 7fc6daadc700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fc6c40b2910 0x7fc6c40b4dd0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:38:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.911+0000 7fc6daadc700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc6d4108870 0x7fc6d419bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:38:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.911+0000 7fc6daadc700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc6d4103780 0x7fc6d41a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:38:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.911+0000 7fc6daadc700 1 --2- 172.21.15.67:0/833578242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc6d4104130 0x7fc6d41a1e40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:38:57.910 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.911+0000 7fc6daadc700 1 -- 172.21.15.67:0/833578242 >> 172.21.15.67:0/833578242 conn(0x7fc6d407a610 msgr2=0x7fc6d41099c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:38:57.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.911+0000 7fc6daadc700 1 -- 172.21.15.67:0/833578242 shutdown_connections 2024-05-03T17:38:57.911 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:38:57.911+0000 7fc6daadc700 1 -- 172.21.15.67:0/833578242 wait complete. 2024-05-03T17:38:57.911 INFO:teuthology.orchestra.run.smithi067.stderr:listed 10 entries 2024-05-03T17:38:58.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:58 smithi082 ceph-mon[33535]: pgmap v846: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:58.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:58 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/833578242' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-03T17:38:58.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:58 smithi070 ceph-mon[33250]: pgmap v846: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:58.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:58 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/833578242' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-03T17:38:58.617 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:58 smithi067 ceph-mon[29731]: pgmap v846: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-03T17:38:58.617 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:58 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/833578242' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-03T17:38:58.650 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-03T17:38:58.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:38:58.651 DEBUG:teuthology.orchestra.run.smithi067:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-03T17:38:58.689 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi067.front.sepia.ceph.com 2024-05-03T17:38:58.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:38:58.690 DEBUG:teuthology.orchestra.run.smithi067:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-03T17:38:58.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:38:58.812 DEBUG:teuthology.orchestra.run.smithi067:> ip netns list 2024-05-03T17:38:58.827 INFO:teuthology.orchestra.run.smithi067.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-03T17:38:58.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:38:58.828 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-03T17:38:58.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-03T17:38:58.894 DEBUG:teuthology.orchestra.run.smithi067:> sudo ip link delete ceph-brx 2024-05-03T17:38:58.997 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-03T17:38:59.008 INFO:tasks.cephadm:Teardown begin 2024-05-03T17:38:59.008 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-03T17:38:59.008 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:38:59.287 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:38:59.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:38:59 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:38:59.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:38:59 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:38:59.554 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:38:59 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-03T17:39:00.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:00 smithi082 ceph-mon[33535]: pgmap v847: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:39:00.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:00 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:39:00.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:00 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:39:00.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:00 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:39:00.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:00 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:39:00.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:00 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:39:00.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:00 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:39:00.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:00 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:39:00.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:00 smithi070 ceph-mon[33250]: pgmap v847: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:39:00.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:00 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:39:00.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:00 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:39:00.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:00 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:39:00.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:00 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:39:00.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:00 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:39:00.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:00 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:39:00.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:00 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:39:00.628 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:00 smithi067 ceph-mon[29731]: pgmap v847: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-03T17:39:00.628 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:00 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:39:00.628 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:00 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:39:00.628 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:00 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:39:00.629 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:00 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-03T17:39:00.629 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:00 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-03T17:39:00.629 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:00 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-03T17:39:00.629 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:00 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' 2024-05-03T17:39:01.047 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.048+0000 7f676b3f9700 1 -- 172.21.15.67:0/2092686041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764101e40 msgr2=0x7f67641022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:01.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.048+0000 7f676b3f9700 1 --2- 172.21.15.67:0/2092686041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764101e40 0x7f67641022c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f6754009990 tx=0x7f6754009ca0 comp rx=0 tx=0).stop 2024-05-03T17:39:01.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.048+0000 7f676b3f9700 1 -- 172.21.15.67:0/2092686041 shutdown_connections 2024-05-03T17:39:01.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.048+0000 7f676b3f9700 1 --2- 172.21.15.67:0/2092686041 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6764102800 0x7f6764110270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:01.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.048+0000 7f676b3f9700 1 --2- 172.21.15.67:0/2092686041 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6764108560 0x7f6764108940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:01.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.048+0000 7f676b3f9700 1 --2- 172.21.15.67:0/2092686041 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764101e40 0x7f67641022c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:01.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.048+0000 7f676b3f9700 1 -- 172.21.15.67:0/2092686041 >> 172.21.15.67:0/2092686041 conn(0x7f67640776c0 msgr2=0x7f6764077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:01.048 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.048+0000 7f676b3f9700 1 -- 172.21.15.67:0/2092686041 shutdown_connections 2024-05-03T17:39:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.048+0000 7f676b3f9700 1 -- 172.21.15.67:0/2092686041 wait complete. 2024-05-03T17:39:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.048+0000 7f676b3f9700 1 Processor -- start 2024-05-03T17:39:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.048+0000 7f676b3f9700 1 -- start start 2024-05-03T17:39:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f676b3f9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6764101e40 0x7f6764103020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f676b3f9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764102800 0x7f6764103560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f676b3f9700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6764108560 0x7f6764103aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f676b3f9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f6764113650 con 0x7f6764102800 2024-05-03T17:39:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f676b3f9700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f67641134d0 con 0x7f6764101e40 2024-05-03T17:39:01.049 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f676b3f9700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f67641137d0 con 0x7f6764108560 2024-05-03T17:39:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f6768994700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764102800 0x7f6764103560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f6768994700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764102800 0x7f6764103560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57554/0 (socket says 172.21.15.67:57554) 2024-05-03T17:39:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f6768994700 1 -- 172.21.15.67:0/2950620373 learned_addr learned my addr 172.21.15.67:0/2950620373 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:39:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f6769195700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6764101e40 0x7f6764103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f6769996700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6764108560 0x7f6764103aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f6768994700 1 -- 172.21.15.67:0/2950620373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6764108560 msgr2=0x7f6764103aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f6768994700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6764108560 0x7f6764103aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:01.050 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f6768994700 1 -- 172.21.15.67:0/2950620373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6764101e40 msgr2=0x7f6764103020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f6768994700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6764101e40 0x7f6764103020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f6768994700 1 -- 172.21.15.67:0/2950620373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67641a8090 con 0x7f6764102800 2024-05-03T17:39:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.049+0000 7f6768994700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764102800 0x7f6764103560 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f67540057c0 tx=0x7f67540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.050+0000 7f675a7fc700 1 -- 172.21.15.67:0/2950620373 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754027070 con 0x7f6764102800 2024-05-03T17:39:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.050+0000 7f675a7fc700 1 -- 172.21.15.67:0/2950620373 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6754003f20 con 0x7f6764102800 2024-05-03T17:39:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.050+0000 7f676b3f9700 1 -- 172.21.15.67:0/2950620373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67641a8320 con 0x7f6764102800 2024-05-03T17:39:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.050+0000 7f675a7fc700 1 -- 172.21.15.67:0/2950620373 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6754022ba0 con 0x7f6764102800 2024-05-03T17:39:01.051 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.050+0000 7f676b3f9700 1 -- 172.21.15.67:0/2950620373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67641a8850 con 0x7f6764102800 2024-05-03T17:39:01.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.051+0000 7f676b3f9700 1 -- 172.21.15.67:0/2950620373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f676406a190 con 0x7f6764102800 2024-05-03T17:39:01.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.053+0000 7f675a7fc700 1 -- 172.21.15.67:0/2950620373 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f675402a070 con 0x7f6764102800 2024-05-03T17:39:01.052 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.053+0000 7f675a7fc700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f67500b2990 0x7f67500b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.054+0000 7f6769195700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f67500b2990 0x7f67500b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.054+0000 7f675a7fc700 1 -- 172.21.15.67:0/2950620373 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f67540e3b80 con 0x7f6764102800 2024-05-03T17:39:01.053 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.054+0000 7f6769195700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f67500b2990 0x7f67500b4e50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6764078c40 tx=0x7f6760007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:01.055 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.057+0000 7f675a7fc700 1 -- 172.21.15.67:0/2950620373 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f6754013070 con 0x7f6764102800 2024-05-03T17:39:01.323 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.325+0000 7f676b3f9700 1 -- 172.21.15.67:0/2950620373 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6764104550 con 0x7f6764102800 2024-05-03T17:39:01.324 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.326+0000 7f675a7fc700 1 -- 172.21.15.67:0/2950620373 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11850 (secure 0 0 0) 0x7f67540ab7c0 con 0x7f6764102800 2024-05-03T17:39:01.325 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:39:01.325 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/316463706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":316463706},{"type":"v1","addr":"172.21.15.67:6839","nonce":316463706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3422086111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3422086111},{"type":"v1","addr":"172.21.15.67:6841","nonce":3422086111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T17:20:39.322107+0000","modified":"2024-05-03T17:23:54.093787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24464,"mds_1":24446,"mds_2":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24421":{"gid":24421,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1568454842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1568454842},{"type":"v1","addr":"172.21.15.82:6835","nonce":1568454842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/370282857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":370282857},{"type":"v1","addr":"172.21.15.82:6837","nonce":370282857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.82:6839/2084218047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2084218047},{"type":"v1","addr":"172.21.15.82:6839","nonce":2084218047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.70:6837/3323713018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3323713018},{"type":"v1","addr":"172.21.15.70:6837","nonce":3323713018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6839/902276547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":902276547},{"type":"v1","addr":"172.21.15.70:6839","nonce":902276547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24442,24446,24464]},"id":1}]} 2024-05-03T17:39:01.326 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.328+0000 7f676b3f9700 1 -- 172.21.15.67:0/2950620373 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f67500b2990 msgr2=0x7f67500b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.328+0000 7f676b3f9700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f67500b2990 0x7f67500b4e50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6764078c40 tx=0x7f6760007400 comp rx=0 tx=0).stop 2024-05-03T17:39:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.328+0000 7f676b3f9700 1 -- 172.21.15.67:0/2950620373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764102800 msgr2=0x7f6764103560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.328+0000 7f676b3f9700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764102800 0x7f6764103560 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f67540057c0 tx=0x7f67540057f0 comp rx=0 tx=0).stop 2024-05-03T17:39:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.328+0000 7f676b3f9700 1 -- 172.21.15.67:0/2950620373 shutdown_connections 2024-05-03T17:39:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.328+0000 7f676b3f9700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f67500b2990 0x7f67500b4e50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.328+0000 7f676b3f9700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f6764108560 0x7f6764103aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.329+0000 7f676b3f9700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f6764101e40 0x7f6764103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:01.327 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.329+0000 7f676b3f9700 1 --2- 172.21.15.67:0/2950620373 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f6764102800 0x7f6764103560 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:01.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.329+0000 7f676b3f9700 1 -- 172.21.15.67:0/2950620373 >> 172.21.15.67:0/2950620373 conn(0x7f67640776c0 msgr2=0x7f6764100b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:01.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.329+0000 7f676b3f9700 1 -- 172.21.15.67:0/2950620373 shutdown_connections 2024-05-03T17:39:01.328 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:01.329+0000 7f676b3f9700 1 -- 172.21.15.67:0/2950620373 wait complete. 2024-05-03T17:39:01.328 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-03T17:39:01.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:01 smithi082 ceph-mon[33535]: Adjusting osd_memory_target on smithi067 to 176.2M 2024-05-03T17:39:01.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:01 smithi082 ceph-mon[33535]: Unable to set osd_memory_target on smithi067 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-05-03T17:39:01.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:01 smithi070 ceph-mon[33250]: Adjusting osd_memory_target on smithi067 to 176.2M 2024-05-03T17:39:01.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:01 smithi070 ceph-mon[33250]: Unable to set osd_memory_target on smithi067 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-05-03T17:39:01.639 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:01 smithi067 ceph-mon[29731]: Adjusting osd_memory_target on smithi067 to 176.2M 2024-05-03T17:39:01.639 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:01 smithi067 ceph-mon[29731]: Unable to set osd_memory_target on smithi067 to 184793907: error parsing value: Value '184793907' is below minimum 939524096 2024-05-03T17:39:01.967 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs dump --format=json 2024-05-03T17:39:02.215 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:39:02.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:02 smithi070 ceph-mon[33250]: pgmap v848: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:39:02.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:02 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2950620373' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:39:02.600 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:02 smithi067 ceph-mon[29731]: pgmap v848: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:39:02.600 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:02 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2950620373' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:39:02.761 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:02 smithi082 ceph-mon[33535]: pgmap v848: 97 pgs: 97 active+clean; 637 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 85 B/s wr, 5 op/s 2024-05-03T17:39:02.761 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:02 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2950620373' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:39:03.366 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:03 smithi067 ceph-mon[29731]: pgmap v849: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-03T17:39:03.761 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:03 smithi082 ceph-mon[33535]: pgmap v849: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-03T17:39:03.805 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:03 smithi070 ceph-mon[33250]: pgmap v849: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2024-05-03T17:39:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.934+0000 7fc53af70700 1 -- 172.21.15.67:0/2457048318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5341076c0 msgr2=0x7fc53410f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.934+0000 7fc53af70700 1 --2- 172.21.15.67:0/2457048318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5341076c0 0x7fc53410f290 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00b0d0 tx=0x7fc52c00b3e0 comp rx=0 tx=0).stop 2024-05-03T17:39:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.934+0000 7fc53af70700 1 -- 172.21.15.67:0/2457048318 shutdown_connections 2024-05-03T17:39:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.934+0000 7fc53af70700 1 --2- 172.21.15.67:0/2457048318 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5341076c0 0x7fc53410f290 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:03.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.934+0000 7fc53af70700 1 --2- 172.21.15.67:0/2457048318 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc534102630 0x7fc534102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.934+0000 7fc53af70700 1 --2- 172.21.15.67:0/2457048318 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc534102fe0 0x7fc534106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.934+0000 7fc53af70700 1 -- 172.21.15.67:0/2457048318 >> 172.21.15.67:0/2457048318 conn(0x7fc534079350 msgr2=0x7fc534079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.934+0000 7fc53af70700 1 -- 172.21.15.67:0/2457048318 shutdown_connections 2024-05-03T17:39:03.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.934+0000 7fc53af70700 1 -- 172.21.15.67:0/2457048318 wait complete. 2024-05-03T17:39:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.935+0000 7fc53af70700 1 Processor -- start 2024-05-03T17:39:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.935+0000 7fc53af70700 1 -- start start 2024-05-03T17:39:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.935+0000 7fc53af70700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc534102630 0x7fc53419bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:03.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.935+0000 7fc53af70700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc534102fe0 0x7fc53419c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.935+0000 7fc53af70700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5341076c0 0x7fc5341a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.935+0000 7fc53af70700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fc5341125b0 con 0x7fc5341076c0 2024-05-03T17:39:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.935+0000 7fc53af70700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fc534112430 con 0x7fc534102fe0 2024-05-03T17:39:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.935+0000 7fc53af70700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fc534112730 con 0x7fc534102630 2024-05-03T17:39:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.935+0000 7fc53950d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5341076c0 0x7fc5341a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.935+0000 7fc53950d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5341076c0 0x7fc5341a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:57572/0 (socket says 172.21.15.67:57572) 2024-05-03T17:39:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.935+0000 7fc53950d700 1 -- 172.21.15.67:0/1464406050 learned_addr learned my addr 172.21.15.67:0/1464406050 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:39:03.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.935+0000 7fc538d0c700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc534102630 0x7fc53419bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.936+0000 7fc533fff700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc534102fe0 0x7fc53419c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.936+0000 7fc53950d700 1 -- 172.21.15.67:0/1464406050 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc534102630 msgr2=0x7fc53419bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.936+0000 7fc53950d700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc534102630 0x7fc53419bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.936+0000 7fc53950d700 1 -- 172.21.15.67:0/1464406050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc534102fe0 msgr2=0x7fc53419c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.936+0000 7fc53950d700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc534102fe0 0x7fc53419c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.936+0000 7fc53950d700 1 -- 172.21.15.67:0/1464406050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5341a0da0 con 0x7fc5341076c0 2024-05-03T17:39:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.936+0000 7fc533fff700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc534102fe0 0x7fc53419c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:39:03.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.936+0000 7fc53950d700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5341076c0 0x7fc5341a0660 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00d010 tx=0x7fc52c0079a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.936+0000 7fc531ffb700 1 -- 172.21.15.67:0/1464406050 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c01f070 con 0x7fc5341076c0 2024-05-03T17:39:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.936+0000 7fc531ffb700 1 -- 172.21.15.67:0/1464406050 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc52c003c00 con 0x7fc5341076c0 2024-05-03T17:39:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.936+0000 7fc53af70700 1 -- 172.21.15.67:0/1464406050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5341a1030 con 0x7fc5341076c0 2024-05-03T17:39:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.936+0000 7fc531ffb700 1 -- 172.21.15.67:0/1464406050 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc52c01b6a0 con 0x7fc5341076c0 2024-05-03T17:39:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.937+0000 7fc53af70700 1 -- 172.21.15.67:0/1464406050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5341a8910 con 0x7fc5341076c0 2024-05-03T17:39:03.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.938+0000 7fc53af70700 1 -- 172.21.15.67:0/1464406050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc534103ac0 con 0x7fc5341076c0 2024-05-03T17:39:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.939+0000 7fc531ffb700 1 -- 172.21.15.67:0/1464406050 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fc52c022070 con 0x7fc5341076c0 2024-05-03T17:39:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.940+0000 7fc531ffb700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fc51c0b2910 0x7fc51c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.940+0000 7fc538d0c700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fc51c0b2910 0x7fc51c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.940+0000 7fc531ffb700 1 -- 172.21.15.67:0/1464406050 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fc52c00f020 con 0x7fc5341076c0 2024-05-03T17:39:03.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.940+0000 7fc538d0c700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fc51c0b2910 0x7fc51c0b4dd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc524005950 tx=0x7fc52400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:03.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:03.943+0000 7fc531ffb700 1 -- 172.21.15.67:0/1464406050 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fc52c029020 con 0x7fc5341076c0 2024-05-03T17:39:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.210+0000 7fc53af70700 1 -- 172.21.15.67:0/1464406050 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc53402e510 con 0x7fc5341076c0 2024-05-03T17:39:04.207 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.211+0000 7fc531ffb700 1 -- 172.21.15.67:0/1464406050 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11850 (secure 0 0 0) 0x7fc52c0ab5b0 con 0x7fc5341076c0 2024-05-03T17:39:04.208 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:39:04.208 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14505,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/1814275265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6834","nonce":1814275265},{"type":"v1","addr":"172.21.15.67:6835","nonce":1814275265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14517,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6837/583931110","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6836","nonce":583931110},{"type":"v1","addr":"172.21.15.67:6837","nonce":583931110}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14565,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6839/316463706","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6838","nonce":316463706},{"type":"v1","addr":"172.21.15.67:6839","nonce":316463706}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6841/3422086111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6840","nonce":3422086111},{"type":"v1","addr":"172.21.15.67:6841","nonce":3422086111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24394,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.82:6833/3927012974","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6832","nonce":3927012974},{"type":"v1","addr":"172.21.15.82:6833","nonce":3927012974}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24398,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.70:6833/2034294041","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6832","nonce":2034294041},{"type":"v1","addr":"172.21.15.70:6833","nonce":2034294041}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-03T17:20:39.322107+0000","modified":"2024-05-03T17:23:54.093787+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24464,"mds_1":24446,"mds_2":24442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24421":{"gid":24421,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6835/1568454842","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6834","nonce":1568454842},{"type":"v1","addr":"172.21.15.82:6835","nonce":1568454842}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24425":{"gid":24425,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.70:6835/1966549782","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6834","nonce":1966549782},{"type":"v1","addr":"172.21.15.70:6835","nonce":1966549782}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.82:6837/370282857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6836","nonce":370282857},{"type":"v1","addr":"172.21.15.82:6837","nonce":370282857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":12,"addr":"172.21.15.82:6839/2084218047","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6838","nonce":2084218047},{"type":"v1","addr":"172.21.15.82:6839","nonce":2084218047}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.70:6837/3323713018","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6836","nonce":3323713018},{"type":"v1","addr":"172.21.15.70:6837","nonce":3323713018}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24464":{"gid":24464,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.70:6839/902276547","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6838","nonce":902276547},{"type":"v1","addr":"172.21.15.70:6839","nonce":902276547}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24464,"qdb_cluster":[24442,24446,24464]},"id":1}]} 2024-05-03T17:39:04.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.213+0000 7fc53af70700 1 -- 172.21.15.67:0/1464406050 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fc51c0b2910 msgr2=0x7fc51c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:04.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.213+0000 7fc53af70700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fc51c0b2910 0x7fc51c0b4dd0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc524005950 tx=0x7fc52400a300 comp rx=0 tx=0).stop 2024-05-03T17:39:04.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.213+0000 7fc53af70700 1 -- 172.21.15.67:0/1464406050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5341076c0 msgr2=0x7fc5341a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:04.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.213+0000 7fc53af70700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5341076c0 0x7fc5341a0660 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fc52c00d010 tx=0x7fc52c0079a0 comp rx=0 tx=0).stop 2024-05-03T17:39:04.210 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.213+0000 7fc53af70700 1 -- 172.21.15.67:0/1464406050 shutdown_connections 2024-05-03T17:39:04.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.213+0000 7fc53af70700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fc51c0b2910 0x7fc51c0b4dd0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:04.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.213+0000 7fc53af70700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fc5341076c0 0x7fc5341a0660 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:04.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.213+0000 7fc53af70700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fc534102630 0x7fc53419bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:04.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.213+0000 7fc53af70700 1 --2- 172.21.15.67:0/1464406050 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fc534102fe0 0x7fc53419c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:04.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.213+0000 7fc53af70700 1 -- 172.21.15.67:0/1464406050 >> 172.21.15.67:0/1464406050 conn(0x7fc534079350 msgr2=0x7fc53410e330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:04.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.213+0000 7fc53af70700 1 -- 172.21.15.67:0/1464406050 shutdown_connections 2024-05-03T17:39:04.211 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:04.213+0000 7fc53af70700 1 -- 172.21.15.67:0/1464406050 wait complete. 2024-05-03T17:39:04.212 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 26 2024-05-03T17:39:04.521 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:04 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1464406050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:39:04.662 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:04 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1464406050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:39:04.706 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:04 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1464406050' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-03T17:39:04.931 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd dump --format=json 2024-05-03T17:39:05.182 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:39:05.576 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:05 smithi067 ceph-mon[29731]: pgmap v850: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-03T17:39:05.761 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:05 smithi082 ceph-mon[33535]: pgmap v850: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-03T17:39:05.806 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:05 smithi070 ceph-mon[33250]: pgmap v850: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2024-05-03T17:39:06.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.929+0000 7f9d0053d700 1 -- 172.21.15.67:0/2682631097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8109530 msgr2=0x7f9cf8109910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:06.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.929+0000 7f9d0053d700 1 --2- 172.21.15.67:0/2682631097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8109530 0x7f9cf8109910 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9cf4009990 tx=0x7f9cf4009ca0 comp rx=0 tx=0).stop 2024-05-03T17:39:06.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.929+0000 7f9d0053d700 1 -- 172.21.15.67:0/2682631097 shutdown_connections 2024-05-03T17:39:06.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.929+0000 7f9d0053d700 1 --2- 172.21.15.67:0/2682631097 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9cf806c590 0x7f9cf8110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.929+0000 7f9d0053d700 1 --2- 172.21.15.67:0/2682631097 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8109530 0x7f9cf8109910 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.929+0000 7f9d0053d700 1 --2- 172.21.15.67:0/2682631097 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cf806bbd0 0x7f9cf806c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.929+0000 7f9d0053d700 1 -- 172.21.15.67:0/2682631097 >> 172.21.15.67:0/2682631097 conn(0x7f9cf8079340 msgr2=0x7f9cf8079750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.929+0000 7f9d0053d700 1 -- 172.21.15.67:0/2682631097 shutdown_connections 2024-05-03T17:39:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.929+0000 7f9d0053d700 1 -- 172.21.15.67:0/2682631097 wait complete. 2024-05-03T17:39:06.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9d0053d700 1 Processor -- start 2024-05-03T17:39:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9d0053d700 1 -- start start 2024-05-03T17:39:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9d0053d700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cf806bbd0 0x7f9cf819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9d0053d700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9cf806c590 0x7f9cf819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9d0053d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8109530 0x7f9cf8199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9d0053d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf8113610 con 0x7f9cf8109530 2024-05-03T17:39:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9d0053d700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf8113490 con 0x7f9cf806bbd0 2024-05-03T17:39:06.938 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9d0053d700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf8113790 con 0x7f9cf806c590 2024-05-03T17:39:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9cfeada700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8109530 0x7f9cf8199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9cfe2d9700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cf806bbd0 0x7f9cf819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9cfeada700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8109530 0x7f9cf8199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55936/0 (socket says 172.21.15.67:55936) 2024-05-03T17:39:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9cfeada700 1 -- 172.21.15.67:0/2340463356 learned_addr learned my addr 172.21.15.67:0/2340463356 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:39:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9cfdad8700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9cf806c590 0x7f9cf819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9cfeada700 1 -- 172.21.15.67:0/2340463356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9cf806c590 msgr2=0x7f9cf819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9cfeada700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9cf806c590 0x7f9cf819fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9cfeada700 1 -- 172.21.15.67:0/2340463356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cf806bbd0 msgr2=0x7f9cf819f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:06.939 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9cfeada700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cf806bbd0 0x7f9cf819f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:06.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.930+0000 7f9cfeada700 1 -- 172.21.15.67:0/2340463356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf819a070 con 0x7f9cf8109530 2024-05-03T17:39:06.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.931+0000 7f9cfeada700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8109530 0x7f9cf8199910 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0007d70 tx=0x7f9cf000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:06.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.931+0000 7f9cef7fe700 1 -- 172.21.15.67:0/2340463356 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0016020 con 0x7f9cf8109530 2024-05-03T17:39:06.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.931+0000 7f9cef7fe700 1 -- 172.21.15.67:0/2340463356 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9cf000ad40 con 0x7f9cf8109530 2024-05-03T17:39:06.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.931+0000 7f9d0053d700 1 -- 172.21.15.67:0/2340463356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cf819a360 con 0x7f9cf8109530 2024-05-03T17:39:06.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.931+0000 7f9cef7fe700 1 -- 172.21.15.67:0/2340463356 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf00115e0 con 0x7f9cf8109530 2024-05-03T17:39:06.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.931+0000 7f9d0053d700 1 -- 172.21.15.67:0/2340463356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cf81a6620 con 0x7f9cf8109530 2024-05-03T17:39:06.940 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.932+0000 7f9d0053d700 1 -- 172.21.15.67:0/2340463356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cf806a190 con 0x7f9cf8109530 2024-05-03T17:39:06.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.933+0000 7f9cef7fe700 1 -- 172.21.15.67:0/2340463356 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f9cf0019020 con 0x7f9cf8109530 2024-05-03T17:39:06.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.934+0000 7f9cef7fe700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9ce40b2910 0x7f9ce40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:06.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.934+0000 7f9cfe2d9700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9ce40b2910 0x7f9ce40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:06.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.934+0000 7f9cef7fe700 1 -- 172.21.15.67:0/2340463356 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f9cf00e2740 con 0x7f9cf8109530 2024-05-03T17:39:06.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.934+0000 7f9cfe2d9700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9ce40b2910 0x7f9ce40b4dd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400b5c0 tx=0x7f9cf4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:06.941 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:06.937+0000 7f9cef7fe700 1 -- 172.21.15.67:0/2340463356 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f9cf0020020 con 0x7f9cf8109530 2024-05-03T17:39:07.153 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.156+0000 7f9d0053d700 1 -- 172.21.15.67:0/2340463356 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9cf802e510 con 0x7f9cf8109530 2024-05-03T17:39:07.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.158+0000 7f9cef7fe700 1 -- 172.21.15.67:0/2340463356 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22635 (secure 0 0 0) 0x7f9cf00aa400 con 0x7f9cf8109530 2024-05-03T17:39:07.155 INFO:teuthology.orchestra.run.smithi067.stdout: 2024-05-03T17:39:07.155 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":81,"fsid":"20c8e22e-0970-11ef-bc97-c7b262605968","created":"2024-05-03T17:12:11.709381+0000","modified":"2024-05-03T17:21:07.584731+0000","last_up_change":"2024-05-03T17:19:29.310797+0000","last_in_change":"2024-05-03T17:19:08.848654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-03T17:15:36.413266+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-03T17:20:31.546843+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-03T17:20:35.261082+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"80962eb1-a506-49bb-a528-dee035eb4322","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6803","nonce":1754365741}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6805","nonce":1754365741}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6809","nonce":1754365741}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":1754365741},{"type":"v1","addr":"172.21.15.67:6807","nonce":1754365741}]},"public_addr":"172.21.15.67:6803/1754365741","cluster_addr":"172.21.15.67:6805/1754365741","heartbeat_back_addr":"172.21.15.67:6809/1754365741","heartbeat_front_addr":"172.21.15.67:6807/1754365741","state":["exists","up"]},{"osd":1,"uuid":"d7721d5f-4373-4cb2-bb57-bc368d6bb818","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6800","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6801","nonce":1006339176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6802","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6803","nonce":1006339176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6806","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6807","nonce":1006339176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6804","nonce":1006339176},{"type":"v1","addr":"172.21.15.70:6805","nonce":1006339176}]},"public_addr":"172.21.15.70:6801/1006339176","cluster_addr":"172.21.15.70:6803/1006339176","heartbeat_back_addr":"172.21.15.70:6807/1006339176","heartbeat_front_addr":"172.21.15.70:6805/1006339176","state":["exists","up"]},{"osd":2,"uuid":"9992fd98-821b-40ce-b051-89befbfd87c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6800","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6801","nonce":1888803264}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6802","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6803","nonce":1888803264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6806","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6807","nonce":1888803264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6804","nonce":1888803264},{"type":"v1","addr":"172.21.15.82:6805","nonce":1888803264}]},"public_addr":"172.21.15.82:6801/1888803264","cluster_addr":"172.21.15.82:6803/1888803264","heartbeat_back_addr":"172.21.15.82:6807/1888803264","heartbeat_front_addr":"172.21.15.82:6805/1888803264","state":["exists","up"]},{"osd":3,"uuid":"14bfe6a6-0bc8-4766-8d36-fbaecc61a396","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6811","nonce":313717723}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6812","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6813","nonce":313717723}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6816","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6817","nonce":313717723}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":313717723},{"type":"v1","addr":"172.21.15.67:6815","nonce":313717723}]},"public_addr":"172.21.15.67:6811/313717723","cluster_addr":"172.21.15.67:6813/313717723","heartbeat_back_addr":"172.21.15.67:6817/313717723","heartbeat_front_addr":"172.21.15.67:6815/313717723","state":["exists","up"]},{"osd":4,"uuid":"5a44aeb8-aabf-4824-be65-2c7170dbee52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6808","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6809","nonce":3526300828}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6810","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6811","nonce":3526300828}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6814","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6815","nonce":3526300828}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6812","nonce":3526300828},{"type":"v1","addr":"172.21.15.70:6813","nonce":3526300828}]},"public_addr":"172.21.15.70:6809/3526300828","cluster_addr":"172.21.15.70:6811/3526300828","heartbeat_back_addr":"172.21.15.70:6815/3526300828","heartbeat_front_addr":"172.21.15.70:6813/3526300828","state":["exists","up"]},{"osd":5,"uuid":"a6ac56df-466e-4096-8fbf-835f5b544297","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6808","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6809","nonce":3754223290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6810","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6811","nonce":3754223290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6814","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6815","nonce":3754223290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6812","nonce":3754223290},{"type":"v1","addr":"172.21.15.82:6813","nonce":3754223290}]},"public_addr":"172.21.15.82:6809/3754223290","cluster_addr":"172.21.15.82:6811/3754223290","heartbeat_back_addr":"172.21.15.82:6815/3754223290","heartbeat_front_addr":"172.21.15.82:6813/3754223290","state":["exists","up"]},{"osd":6,"uuid":"4d8a5e8a-66ac-462e-b960-24c0ee248fac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6819","nonce":4033769161}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6820","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6821","nonce":4033769161}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6825","nonce":4033769161}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":4033769161},{"type":"v1","addr":"172.21.15.67:6823","nonce":4033769161}]},"public_addr":"172.21.15.67:6819/4033769161","cluster_addr":"172.21.15.67:6821/4033769161","heartbeat_back_addr":"172.21.15.67:6825/4033769161","heartbeat_front_addr":"172.21.15.67:6823/4033769161","state":["exists","up"]},{"osd":7,"uuid":"eb983837-8172-4802-9843-6360cda8f4be","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6816","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6817","nonce":3366146173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6818","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6819","nonce":3366146173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6822","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6823","nonce":3366146173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6820","nonce":3366146173},{"type":"v1","addr":"172.21.15.70:6821","nonce":3366146173}]},"public_addr":"172.21.15.70:6817/3366146173","cluster_addr":"172.21.15.70:6819/3366146173","heartbeat_back_addr":"172.21.15.70:6823/3366146173","heartbeat_front_addr":"172.21.15.70:6821/3366146173","state":["exists","up"]},{"osd":8,"uuid":"c9d4ce56-e940-48d1-a9b5-437139e3fed0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6816","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6817","nonce":347699286}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6818","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6819","nonce":347699286}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6822","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6823","nonce":347699286}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6820","nonce":347699286},{"type":"v1","addr":"172.21.15.82:6821","nonce":347699286}]},"public_addr":"172.21.15.82:6817/347699286","cluster_addr":"172.21.15.82:6819/347699286","heartbeat_back_addr":"172.21.15.82:6823/347699286","heartbeat_front_addr":"172.21.15.82:6821/347699286","state":["exists","up"]},{"osd":9,"uuid":"09596d19-f2d7-485e-a028-9ce5468c6651","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6827","nonce":2600095198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6829","nonce":2600095198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6833","nonce":2600095198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":2600095198},{"type":"v1","addr":"172.21.15.67:6831","nonce":2600095198}]},"public_addr":"172.21.15.67:6827/2600095198","cluster_addr":"172.21.15.67:6829/2600095198","heartbeat_back_addr":"172.21.15.67:6833/2600095198","heartbeat_front_addr":"172.21.15.67:6831/2600095198","state":["exists","up"]},{"osd":10,"uuid":"ec4cffbc-0127-47e5-8176-0972f74a5514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6824","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6825","nonce":2897583629}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6826","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6827","nonce":2897583629}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6830","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6831","nonce":2897583629}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.70:6828","nonce":2897583629},{"type":"v1","addr":"172.21.15.70:6829","nonce":2897583629}]},"public_addr":"172.21.15.70:6825/2897583629","cluster_addr":"172.21.15.70:6827/2897583629","heartbeat_back_addr":"172.21.15.70:6831/2897583629","heartbeat_front_addr":"172.21.15.70:6829/2897583629","state":["exists","up"]},{"osd":11,"uuid":"a8acfb0b-98e8-49e1-a114-f33aa8160b62","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6824","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6825","nonce":4082656149}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6826","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6827","nonce":4082656149}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6830","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6831","nonce":4082656149}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.82:6828","nonce":4082656149},{"type":"v1","addr":"172.21.15.82:6829","nonce":4082656149}]},"public_addr":"172.21.15.82:6825/4082656149","cluster_addr":"172.21.15.82:6827/4082656149","heartbeat_back_addr":"172.21.15.82:6831/4082656149","heartbeat_front_addr":"172.21.15.82:6829/4082656149","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:14:45.596665+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:09.538408+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:33.608945+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:15:59.803102+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:25.187966+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:16:49.892160+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:16.575973+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:17:42.115955+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:07.655558+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:18:35.356779+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:01.144695+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-03T17:19:27.577513+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.b","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"3.17","mappings":[{"from":4,"to":8}]},{"pgid":"3.19","mappings":[{"from":3,"to":7}]},{"pgid":"3.1d","mappings":[{"from":5,"to":9}]},{"pgid":"3.24","mappings":[{"from":4,"to":9}]},{"pgid":"3.33","mappings":[{"from":5,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.67:0/2131289004":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/3335149382":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/992151181":"2024-05-04T17:13:06.324002+0000","172.21.15.67:6801/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/1345903711":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2614686067":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/3137329076":"2024-05-04T17:12:38.899429+0000","172.21.15.67:0/2323686292":"2024-05-04T17:13:06.324002+0000","172.21.15.67:0/2579128777":"2024-05-04T17:12:38.899429+0000","172.21.15.67:6800/3335149382":"2024-05-04T17:13:06.324002+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-03T17:39:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.160+0000 7f9d0053d700 1 -- 172.21.15.67:0/2340463356 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9ce40b2910 msgr2=0x7f9ce40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.160+0000 7f9d0053d700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9ce40b2910 0x7f9ce40b4dd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9cf400b5c0 tx=0x7f9cf4005960 comp rx=0 tx=0).stop 2024-05-03T17:39:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.160+0000 7f9d0053d700 1 -- 172.21.15.67:0/2340463356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8109530 msgr2=0x7f9cf8199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.160+0000 7f9d0053d700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8109530 0x7f9cf8199910 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0007d70 tx=0x7f9cf000a340 comp rx=0 tx=0).stop 2024-05-03T17:39:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.160+0000 7f9d0053d700 1 -- 172.21.15.67:0/2340463356 shutdown_connections 2024-05-03T17:39:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.160+0000 7f9d0053d700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f9ce40b2910 0x7f9ce40b4dd0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.160+0000 7f9d0053d700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9cf8109530 0x7f9cf8199910 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:07.157 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.160+0000 7f9d0053d700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f9cf806bbd0 0x7f9cf819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.160+0000 7f9d0053d700 1 --2- 172.21.15.67:0/2340463356 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f9cf806c590 0x7f9cf819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.160+0000 7f9d0053d700 1 -- 172.21.15.67:0/2340463356 >> 172.21.15.67:0/2340463356 conn(0x7f9cf8079340 msgr2=0x7f9cf8100fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.160+0000 7f9d0053d700 1 -- 172.21.15.67:0/2340463356 shutdown_connections 2024-05-03T17:39:07.158 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:07.160+0000 7f9d0053d700 1 -- 172.21.15.67:0/2340463356 wait complete. 2024-05-03T17:39:07.869 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs fail cephfs 2024-05-03T17:39:08.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:07 smithi070 ceph-mon[33250]: pgmap v851: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-03T17:39:08.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:07 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2340463356' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:39:08.115 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:39:08.199 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:07 smithi067 ceph-mon[29731]: pgmap v851: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-03T17:39:08.199 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:07 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2340463356' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:39:08.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:07 smithi082 ceph-mon[33535]: pgmap v851: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-05-03T17:39:08.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:07 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2340463356' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-03T17:39:09.855 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:09 smithi067 ceph-mon[29731]: pgmap v852: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-03T17:39:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.863+0000 7fcd6548b700 1 -- 172.21.15.67:0/1477048866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd60103ef0 msgr2=0x7fcd601103a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.863+0000 7fcd6548b700 1 --2- 172.21.15.67:0/1477048866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd60103ef0 0x7fcd601103a0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fcd500099c0 tx=0x7fcd50009cd0 comp rx=0 tx=0).stop 2024-05-03T17:39:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.863+0000 7fcd6548b700 1 -- 172.21.15.67:0/1477048866 shutdown_connections 2024-05-03T17:39:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.863+0000 7fcd6548b700 1 --2- 172.21.15.67:0/1477048866 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd60103ef0 0x7fcd601103a0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.863+0000 7fcd6548b700 1 --2- 172.21.15.67:0/1477048866 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd601095d0 0x7fcd601099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.863+0000 7fcd6548b700 1 --2- 172.21.15.67:0/1477048866 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd60103530 0x7fcd601039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:09.860 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.863+0000 7fcd6548b700 1 -- 172.21.15.67:0/1477048866 >> 172.21.15.67:0/1477048866 conn(0x7fcd6007a690 msgr2=0x7fcd6007aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:09.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.863+0000 7fcd6548b700 1 -- 172.21.15.67:0/1477048866 shutdown_connections 2024-05-03T17:39:09.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.863+0000 7fcd6548b700 1 -- 172.21.15.67:0/1477048866 wait complete. 2024-05-03T17:39:09.861 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.863+0000 7fcd6548b700 1 Processor -- start 2024-05-03T17:39:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd6548b700 1 -- start start 2024-05-03T17:39:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd6548b700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd60103530 0x7fcd6019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd6548b700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd60103ef0 0x7fcd6019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd6548b700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd601095d0 0x7fcd60199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd6548b700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7fcd60113730 con 0x7fcd601095d0 2024-05-03T17:39:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd6548b700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7fcd601135b0 con 0x7fcd60103530 2024-05-03T17:39:09.863 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd6548b700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7fcd601138b0 con 0x7fcd60103ef0 2024-05-03T17:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd5f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd601095d0 0x7fcd60199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd5f7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd601095d0 0x7fcd60199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55954/0 (socket says 172.21.15.67:55954) 2024-05-03T17:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd5f7fe700 1 -- 172.21.15.67:0/3154372746 learned_addr learned my addr 172.21.15.67:0/3154372746 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd5e7fc700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd60103ef0 0x7fcd6019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd5effd700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd60103530 0x7fcd6019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:09.864 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd5f7fe700 1 -- 172.21.15.67:0/3154372746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd60103ef0 msgr2=0x7fcd6019fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd5f7fe700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd60103ef0 0x7fcd6019fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd5f7fe700 1 -- 172.21.15.67:0/3154372746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd60103530 msgr2=0x7fcd6019f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd5f7fe700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd60103530 0x7fcd6019f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd5f7fe700 1 -- 172.21.15.67:0/3154372746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd6019a0f0 con 0x7fcd601095d0 2024-05-03T17:39:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.864+0000 7fcd5f7fe700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd601095d0 0x7fcd60199990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009990 tx=0x7fcd500047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.865+0000 7fcd57fff700 1 -- 172.21.15.67:0/3154372746 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd50027070 con 0x7fcd601095d0 2024-05-03T17:39:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.865+0000 7fcd6548b700 1 -- 172.21.15.67:0/3154372746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd6019a380 con 0x7fcd601095d0 2024-05-03T17:39:09.865 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.865+0000 7fcd57fff700 1 -- 172.21.15.67:0/3154372746 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcd50004020 con 0x7fcd601095d0 2024-05-03T17:39:09.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.865+0000 7fcd57fff700 1 -- 172.21.15.67:0/3154372746 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd50022ce0 con 0x7fcd601095d0 2024-05-03T17:39:09.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.865+0000 7fcd6548b700 1 -- 172.21.15.67:0/3154372746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd601a6650 con 0x7fcd601095d0 2024-05-03T17:39:09.866 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.866+0000 7fcd6548b700 1 -- 172.21.15.67:0/3154372746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd60104c40 con 0x7fcd601095d0 2024-05-03T17:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.867+0000 7fcd57fff700 1 -- 172.21.15.67:0/3154372746 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7fcd5002a070 con 0x7fcd601095d0 2024-05-03T17:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.868+0000 7fcd57fff700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fcd4c0b2910 0x7fcd4c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.868+0000 7fcd5effd700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fcd4c0b2910 0x7fcd4c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.868+0000 7fcd57fff700 1 -- 172.21.15.67:0/3154372746 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fcd500e3850 con 0x7fcd601095d0 2024-05-03T17:39:09.867 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.868+0000 7fcd5effd700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fcd4c0b2910 0x7fcd4c0b4dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcd60109070 tx=0x7fcd58009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:09.868 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:09.872+0000 7fcd57fff700 1 -- 172.21.15.67:0/3154372746 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7fcd50031070 con 0x7fcd601095d0 2024-05-03T17:39:10.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:09 smithi070 ceph-mon[33250]: pgmap v852: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-03T17:39:10.130 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.133+0000 7fcd6548b700 1 -- 172.21.15.67:0/3154372746 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fcd6002e510 con 0x7fcd601095d0 2024-05-03T17:39:10.132 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.136+0000 7fcd57fff700 1 -- 172.21.15.67:0/3154372746 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) ==== 161+0+0 (secure 0 0 0) 0x7fcd50013070 con 0x7fcd601095d0 2024-05-03T17:39:10.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.139+0000 7fcd6548b700 1 -- 172.21.15.67:0/3154372746 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fcd4c0b2910 msgr2=0x7fcd4c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:10.136 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.139+0000 7fcd6548b700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fcd4c0b2910 0x7fcd4c0b4dd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fcd60109070 tx=0x7fcd58009040 comp rx=0 tx=0).stop 2024-05-03T17:39:10.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.139+0000 7fcd6548b700 1 -- 172.21.15.67:0/3154372746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd601095d0 msgr2=0x7fcd60199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:10.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.139+0000 7fcd6548b700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd601095d0 0x7fcd60199990 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fcd50009990 tx=0x7fcd500047b0 comp rx=0 tx=0).stop 2024-05-03T17:39:10.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.139+0000 7fcd6548b700 1 -- 172.21.15.67:0/3154372746 shutdown_connections 2024-05-03T17:39:10.137 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.139+0000 7fcd6548b700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7fcd4c0b2910 0x7fcd4c0b4dd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:10.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.139+0000 7fcd6548b700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcd601095d0 0x7fcd60199990 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:10.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.139+0000 7fcd6548b700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7fcd60103530 0x7fcd6019f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:10.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.139+0000 7fcd6548b700 1 --2- 172.21.15.67:0/3154372746 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7fcd60103ef0 0x7fcd6019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:10.138 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.139+0000 7fcd6548b700 1 -- 172.21.15.67:0/3154372746 >> 172.21.15.67:0/3154372746 conn(0x7fcd6007a690 msgr2=0x7fcd60102130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:10.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.139+0000 7fcd6548b700 1 -- 172.21.15.67:0/3154372746 shutdown_connections 2024-05-03T17:39:10.139 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:10.139+0000 7fcd6548b700 1 -- 172.21.15.67:0/3154372746 wait complete. 2024-05-03T17:39:10.140 INFO:teuthology.orchestra.run.smithi067.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-03T17:39:10.177 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:09 smithi082 ceph-mon[33535]: pgmap v852: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-03T17:39:10.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:10 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a[29727]: 2024-05-03T17:39:10.134+0000 7f820535b700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-03T17:39:10.511 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:39:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-f[64389]: ignoring --setuser ceph since I am not root 2024-05-03T17:39:10.511 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:39:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-f[64389]: ignoring --setgroup ceph since I am not root 2024-05-03T17:39:10.511 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:39:10 smithi082 ceph-mds[64393]: -- 172.21.15.82:0/3014947357 <== mon.1 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aea45b7a40 con 0x55aea458ac00 2024-05-03T17:39:10.511 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:39:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-f[64389]: starting mds.f at 2024-05-03T17:39:10.512 INFO:journalctl@ceph.mds.c.smithi082.stdout:May 03 17:39:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-c[64699]: ignoring --setuser ceph since I am not root 2024-05-03T17:39:10.512 INFO:journalctl@ceph.mds.c.smithi082.stdout:May 03 17:39:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-c[64699]: ignoring --setgroup ceph since I am not root 2024-05-03T17:39:10.512 INFO:journalctl@ceph.mds.c.smithi082.stdout:May 03 17:39:10 smithi082 ceph-mds[64703]: -- 172.21.15.82:0/2297050264 <== mon.1 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55791d0c1a40 con 0x55791d094c00 2024-05-03T17:39:10.512 INFO:journalctl@ceph.mds.c.smithi082.stdout:May 03 17:39:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-c[64699]: starting mds.c at 2024-05-03T17:39:10.512 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:39:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-i[65012]: ignoring --setuser ceph since I am not root 2024-05-03T17:39:10.512 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:39:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-i[65012]: ignoring --setgroup ceph since I am not root 2024-05-03T17:39:10.513 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:39:10 smithi082 ceph-mds[65016]: -- 172.21.15.82:0/3732898278 <== mon.0 v2:172.21.15.67:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5573f0869a40 con 0x5573f083c800 2024-05-03T17:39:10.513 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:39:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-i[65012]: starting mds.i at 2024-05-03T17:39:10.554 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:39:10 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b[63855]: ignoring --setuser ceph since I am not root 2024-05-03T17:39:10.555 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:39:10 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b[63855]: ignoring --setgroup ceph since I am not root 2024-05-03T17:39:10.555 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:39:10 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b[63855]: starting mds.b at 2024-05-03T17:39:10.555 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:39:10 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-h[63535]: ignoring --setuser ceph since I am not root 2024-05-03T17:39:10.555 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:39:10 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-h[63535]: ignoring --setgroup ceph since I am not root 2024-05-03T17:39:10.555 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:39:10 smithi070 ceph-mds[63539]: -- 172.21.15.70:0/1423608256 <== mon.1 v2:172.21.15.82:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564ade6a5a40 con 0x564ade678c00 2024-05-03T17:39:10.555 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:39:10 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-h[63535]: starting mds.h at 2024-05-03T17:39:10.935 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:10 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3154372746' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-03T17:39:10.935 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:10 smithi067 ceph-mon[29731]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-03T17:39:10.935 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:10 smithi067 ceph-mon[29731]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-03T17:39:10.935 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:10 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/3154372746' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-03T17:39:10.936 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:10 smithi067 ceph-mon[29731]: osdmap e82: 12 total, 12 up, 12 in 2024-05-03T17:39:10.936 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:10 smithi067 ceph-mon[29731]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-03T17:39:10.970 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-05-03T17:39:11.054 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:10 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3154372746' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-03T17:39:11.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:10 smithi070 ceph-mon[33250]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-03T17:39:11.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:10 smithi070 ceph-mon[33250]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-03T17:39:11.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:10 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/3154372746' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-03T17:39:11.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:10 smithi070 ceph-mon[33250]: osdmap e82: 12 total, 12 up, 12 in 2024-05-03T17:39:11.055 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:10 smithi070 ceph-mon[33250]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-03T17:39:11.229 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:39:11.260 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:10 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3154372746' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-03T17:39:11.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:10 smithi082 ceph-mon[33535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-03T17:39:11.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:10 smithi082 ceph-mon[33535]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-03T17:39:11.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:10 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/3154372746' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-03T17:39:11.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:10 smithi082 ceph-mon[33535]: osdmap e82: 12 total, 12 up, 12 in 2024-05-03T17:39:11.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:10 smithi082 ceph-mon[33535]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-03T17:39:12.427 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: pgmap v854: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 6 op/s 2024-05-03T17:39:12.427 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.70:6834/1707325821,v1:172.21.15.70:6837/1707325821] up:boot 2024-05-03T17:39:12.427 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.70:6835/807270972,v1:172.21.15.70:6838/807270972] up:boot 2024-05-03T17:39:12.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.70:6836/1310418312,v1:172.21.15.70:6839/1310418312] up:boot 2024-05-03T17:39:12.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.82:6834/3304266262,v1:172.21.15.82:6835/3304266262] up:boot 2024-05-03T17:39:12.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.82:6836/1188453829,v1:172.21.15.82:6838/1188453829] up:boot 2024-05-03T17:39:12.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: mds.? [v2:172.21.15.82:6837/1723696544,v1:172.21.15.82:6839/1723696544] up:boot 2024-05-03T17:39:12.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-05-03T17:39:12.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-03T17:39:12.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-03T17:39:12.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-03T17:39:12.428 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-03T17:39:12.429 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-03T17:39:12.429 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:12 smithi067 ceph-mon[29731]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-03T17:39:12.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: pgmap v854: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 6 op/s 2024-05-03T17:39:12.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.70:6834/1707325821,v1:172.21.15.70:6837/1707325821] up:boot 2024-05-03T17:39:12.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.70:6835/807270972,v1:172.21.15.70:6838/807270972] up:boot 2024-05-03T17:39:12.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.70:6836/1310418312,v1:172.21.15.70:6839/1310418312] up:boot 2024-05-03T17:39:12.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.82:6834/3304266262,v1:172.21.15.82:6835/3304266262] up:boot 2024-05-03T17:39:12.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.82:6836/1188453829,v1:172.21.15.82:6838/1188453829] up:boot 2024-05-03T17:39:12.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: mds.? [v2:172.21.15.82:6837/1723696544,v1:172.21.15.82:6839/1723696544] up:boot 2024-05-03T17:39:12.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-05-03T17:39:12.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-03T17:39:12.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-03T17:39:12.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-03T17:39:12.512 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-03T17:39:12.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-03T17:39:12.513 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:12 smithi082 ceph-mon[33535]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-03T17:39:12.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: pgmap v854: 97 pgs: 97 active+clean; 638 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 6 op/s 2024-05-03T17:39:12.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.70:6834/1707325821,v1:172.21.15.70:6837/1707325821] up:boot 2024-05-03T17:39:12.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.70:6835/807270972,v1:172.21.15.70:6838/807270972] up:boot 2024-05-03T17:39:12.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.70:6836/1310418312,v1:172.21.15.70:6839/1310418312] up:boot 2024-05-03T17:39:12.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.82:6834/3304266262,v1:172.21.15.82:6835/3304266262] up:boot 2024-05-03T17:39:12.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.82:6836/1188453829,v1:172.21.15.82:6838/1188453829] up:boot 2024-05-03T17:39:12.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: mds.? [v2:172.21.15.82:6837/1723696544,v1:172.21.15.82:6839/1723696544] up:boot 2024-05-03T17:39:12.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: fsmap cephfs:0/3 12 up:standby, 3 failed 2024-05-03T17:39:12.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-03T17:39:12.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-03T17:39:12.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-03T17:39:12.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-03T17:39:12.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-03T17:39:12.556 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:12 smithi070 ceph-mon[33250]: from='mgr.14152 172.21.15.67:0/2898261166' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-03T17:39:13.029 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.033+0000 7f19c9266700 1 -- 172.21.15.67:0/715212700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c4102820 msgr2=0x7f19c4110290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:13.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.033+0000 7f19c9266700 1 --2- 172.21.15.67:0/715212700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c4102820 0x7f19c4110290 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f19bc00b0a0 tx=0x7f19bc00b3b0 comp rx=0 tx=0).stop 2024-05-03T17:39:13.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.033+0000 7f19c9266700 1 -- 172.21.15.67:0/715212700 shutdown_connections 2024-05-03T17:39:13.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.033+0000 7f19c9266700 1 --2- 172.21.15.67:0/715212700 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c4102820 0x7f19c4110290 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:13.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.033+0000 7f19c9266700 1 --2- 172.21.15.67:0/715212700 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19c4108580 0x7f19c4108960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:13.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.033+0000 7f19c9266700 1 --2- 172.21.15.67:0/715212700 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19c4101e60 0x7f19c41022e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:13.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.033+0000 7f19c9266700 1 -- 172.21.15.67:0/715212700 >> 172.21.15.67:0/715212700 conn(0x7f19c40776c0 msgr2=0x7f19c4077ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:13.030 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.033+0000 7f19c9266700 1 -- 172.21.15.67:0/715212700 shutdown_connections 2024-05-03T17:39:13.031 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.033+0000 7f19c9266700 1 -- 172.21.15.67:0/715212700 wait complete. 2024-05-03T17:39:13.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.033+0000 7f19c9266700 1 Processor -- start 2024-05-03T17:39:13.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.033+0000 7f19c9266700 1 -- start start 2024-05-03T17:39:13.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c9266700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c4101e60 0x7f19c4103040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:13.032 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c9266700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19c4102820 0x7f19c4103580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:13.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c9266700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19c4108580 0x7f19c4103ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:13.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c9266700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f19c4113690 con 0x7f19c4101e60 2024-05-03T17:39:13.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c9266700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f19c4113510 con 0x7f19c4108580 2024-05-03T17:39:13.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c9266700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f19c4113810 con 0x7f19c4102820 2024-05-03T17:39:13.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c4101e60 0x7f19c4103040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:13.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c2ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c4101e60 0x7f19c4103040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:55972/0 (socket says 172.21.15.67:55972) 2024-05-03T17:39:13.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c2ffd700 1 -- 172.21.15.67:0/1763152253 learned_addr learned my addr 172.21.15.67:0/1763152253 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:39:13.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c27fc700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19c4102820 0x7f19c4103580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:13.033 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c37fe700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19c4108580 0x7f19c4103ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c2ffd700 1 -- 172.21.15.67:0/1763152253 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19c4102820 msgr2=0x7f19c4103580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c2ffd700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19c4102820 0x7f19c4103580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c2ffd700 1 -- 172.21.15.67:0/1763152253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19c4108580 msgr2=0x7f19c4103ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c2ffd700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19c4108580 0x7f19c4103ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c2ffd700 1 -- 172.21.15.67:0/1763152253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19c41a8020 con 0x7f19c4101e60 2024-05-03T17:39:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.034+0000 7f19c2ffd700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c4101e60 0x7f19c4103040 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f19b400d980 tx=0x7f19b400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.035+0000 7f19abfff700 1 -- 172.21.15.67:0/1763152253 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b4017070 con 0x7f19c4101e60 2024-05-03T17:39:13.034 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.035+0000 7f19c9266700 1 -- 172.21.15.67:0/1763152253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19c41a8310 con 0x7f19c4101e60 2024-05-03T17:39:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.035+0000 7f19abfff700 1 -- 172.21.15.67:0/1763152253 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f19b400fd40 con 0x7f19c4101e60 2024-05-03T17:39:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.035+0000 7f19abfff700 1 -- 172.21.15.67:0/1763152253 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b400e3b0 con 0x7f19c4101e60 2024-05-03T17:39:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.035+0000 7f19c9266700 1 -- 172.21.15.67:0/1763152253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19c41a8850 con 0x7f19c4101e60 2024-05-03T17:39:13.035 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.036+0000 7f19c9266700 1 -- 172.21.15.67:0/1763152253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19c406a190 con 0x7f19c4101e60 2024-05-03T17:39:13.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.037+0000 7f19abfff700 1 -- 172.21.15.67:0/1763152253 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f19b400f890 con 0x7f19c4101e60 2024-05-03T17:39:13.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.038+0000 7f19abfff700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f19ac0b2910 0x7f19ac0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:13.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.038+0000 7f19c27fc700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f19ac0b2910 0x7f19ac0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:13.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.038+0000 7f19abfff700 1 -- 172.21.15.67:0/1763152253 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8506+0+0 (secure 0 0 0) 0x7f19b40e1f30 con 0x7f19c4101e60 2024-05-03T17:39:13.036 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.038+0000 7f19c27fc700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f19ac0b2910 0x7f19ac0b4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f19c4077ce0 tx=0x7f19b800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:13.039 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.042+0000 7f19abfff700 1 -- 172.21.15.67:0/1763152253 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f19b40a99e0 con 0x7f19c4101e60 2024-05-03T17:39:13.300 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:13.304+0000 7f19c9266700 1 -- 172.21.15.67:0/1763152253 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f19c402e510 con 0x7f19c4101e60 2024-05-03T17:39:14.144 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.147+0000 7f19abfff700 1 -- 172.21.15.67:0/1763152253 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v29) ==== 104+0+0 (secure 0 0 0) 0x7f19b40a92e0 con 0x7f19c4101e60 2024-05-03T17:39:14.148 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.151+0000 7f19c9266700 1 -- 172.21.15.67:0/1763152253 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f19ac0b2910 msgr2=0x7f19ac0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:14.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.151+0000 7f19c9266700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f19ac0b2910 0x7f19ac0b4dd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f19c4077ce0 tx=0x7f19b800a380 comp rx=0 tx=0).stop 2024-05-03T17:39:14.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.151+0000 7f19c9266700 1 -- 172.21.15.67:0/1763152253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c4101e60 msgr2=0x7f19c4103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:14.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.151+0000 7f19c9266700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c4101e60 0x7f19c4103040 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f19b400d980 tx=0x7f19b400dc90 comp rx=0 tx=0).stop 2024-05-03T17:39:14.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.152+0000 7f19c9266700 1 -- 172.21.15.67:0/1763152253 shutdown_connections 2024-05-03T17:39:14.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.152+0000 7f19c9266700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f19ac0b2910 0x7f19ac0b4dd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:14.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.152+0000 7f19c9266700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f19c4108580 0x7f19c4103ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:14.149 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.152+0000 7f19c9266700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f19c4101e60 0x7f19c4103040 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:14.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.152+0000 7f19c9266700 1 --2- 172.21.15.67:0/1763152253 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f19c4102820 0x7f19c4103580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:14.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.152+0000 7f19c9266700 1 -- 172.21.15.67:0/1763152253 >> 172.21.15.67:0/1763152253 conn(0x7f19c40776c0 msgr2=0x7f19c4100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:14.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.152+0000 7f19c9266700 1 -- 172.21.15.67:0/1763152253 shutdown_connections 2024-05-03T17:39:14.150 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:14.152+0000 7f19c9266700 1 -- 172.21.15.67:0/1763152253 wait complete. 2024-05-03T17:39:14.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:14 smithi067 ceph-mon[29731]: pgmap v855: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 102 B/s wr, 4 op/s 2024-05-03T17:39:14.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:14 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1763152253' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T17:39:14.510 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:14 smithi082 ceph-mon[33535]: pgmap v855: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 102 B/s wr, 4 op/s 2024-05-03T17:39:14.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:14 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1763152253' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T17:39:14.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:14 smithi070 ceph-mon[33250]: pgmap v855: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 102 B/s wr, 4 op/s 2024-05-03T17:39:14.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:14 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1763152253' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-03T17:39:14.853 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-03T17:39:15.106 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:39:15.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:15 smithi067 ceph-mon[29731]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-03T17:39:15.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:15 smithi067 ceph-mon[29731]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-03T17:39:15.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:15 smithi067 ceph-mon[29731]: Cluster is now healthy 2024-05-03T17:39:15.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:15 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1763152253' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-03T17:39:15.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:15 smithi067 ceph-mon[29731]: fsmap 12 up:standby 2024-05-03T17:39:15.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:15 smithi082 ceph-mon[33535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-03T17:39:15.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:15 smithi082 ceph-mon[33535]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-03T17:39:15.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:15 smithi082 ceph-mon[33535]: Cluster is now healthy 2024-05-03T17:39:15.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:15 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1763152253' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-03T17:39:15.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:15 smithi082 ceph-mon[33535]: fsmap 12 up:standby 2024-05-03T17:39:15.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:15 smithi070 ceph-mon[33250]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-03T17:39:15.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:15 smithi070 ceph-mon[33250]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-03T17:39:15.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:15 smithi070 ceph-mon[33250]: Cluster is now healthy 2024-05-03T17:39:15.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:15 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1763152253' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-03T17:39:15.555 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:15 smithi070 ceph-mon[33250]: fsmap 12 up:standby 2024-05-03T17:39:16.413 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:16 smithi067 ceph-mon[29731]: pgmap v856: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-05-03T17:39:16.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:16 smithi082 ceph-mon[33535]: pgmap v856: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-05-03T17:39:16.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:16 smithi070 ceph-mon[33250]: pgmap v856: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-05-03T17:39:16.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.996+0000 7f35eead8700 1 -- 172.21.15.67:0/703478206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e81076c0 msgr2=0x7f35e810f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:16.995 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.996+0000 7f35eead8700 1 --2- 172.21.15.67:0/703478206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e81076c0 0x7f35e810f290 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f35e000b0d0 tx=0x7f35e000b3e0 comp rx=0 tx=0).stop 2024-05-03T17:39:16.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.996+0000 7f35eead8700 1 -- 172.21.15.67:0/703478206 shutdown_connections 2024-05-03T17:39:16.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.996+0000 7f35eead8700 1 --2- 172.21.15.67:0/703478206 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e81076c0 0x7f35e810f290 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:16.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.996+0000 7f35eead8700 1 --2- 172.21.15.67:0/703478206 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35e8102630 0x7f35e8102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:16.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.996+0000 7f35eead8700 1 --2- 172.21.15.67:0/703478206 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35e8102fe0 0x7f35e8106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:16.996 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.996+0000 7f35eead8700 1 -- 172.21.15.67:0/703478206 >> 172.21.15.67:0/703478206 conn(0x7f35e8079350 msgr2=0x7f35e8079760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:16.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.996+0000 7f35eead8700 1 -- 172.21.15.67:0/703478206 shutdown_connections 2024-05-03T17:39:16.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.996+0000 7f35eead8700 1 -- 172.21.15.67:0/703478206 wait complete. 2024-05-03T17:39:16.997 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.996+0000 7f35eead8700 1 Processor -- start 2024-05-03T17:39:16.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.996+0000 7f35eead8700 1 -- start start 2024-05-03T17:39:16.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35eead8700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35e8102630 0x7f35e8104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:16.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35eead8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e8102fe0 0x7f35e81045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:16.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35eead8700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35e81076c0 0x7f35e8104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:16.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35eead8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f35e8112560 con 0x7f35e8102fe0 2024-05-03T17:39:16.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35eead8700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f35e81123e0 con 0x7f35e81076c0 2024-05-03T17:39:16.998 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35eead8700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f35e81126e0 con 0x7f35e8102630 2024-05-03T17:39:16.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35e7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e8102fe0 0x7f35e81045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:16.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35e7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e8102fe0 0x7f35e81045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45852/0 (socket says 172.21.15.67:45852) 2024-05-03T17:39:16.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35e7fff700 1 -- 172.21.15.67:0/2844888444 learned_addr learned my addr 172.21.15.67:0/2844888444 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:39:16.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35ec874700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35e8102630 0x7f35e8104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:16.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35ed075700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35e81076c0 0x7f35e8104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:16.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35e7fff700 1 -- 172.21.15.67:0/2844888444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35e8102630 msgr2=0x7f35e8104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:16.999 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35e7fff700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35e8102630 0x7f35e8104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:17.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35e7fff700 1 -- 172.21.15.67:0/2844888444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35e81076c0 msgr2=0x7f35e8104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:17.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35e7fff700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35e81076c0 0x7f35e8104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:17.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35e7fff700 1 -- 172.21.15.67:0/2844888444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e81a8080 con 0x7f35e8102fe0 2024-05-03T17:39:17.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.997+0000 7f35e7fff700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e8102fe0 0x7f35e81045c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f35d400d980 tx=0x7f35d400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:17.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.998+0000 7f35e5ffb700 1 -- 172.21.15.67:0/2844888444 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d4017070 con 0x7f35e8102fe0 2024-05-03T17:39:17.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.998+0000 7f35e5ffb700 1 -- 172.21.15.67:0/2844888444 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f35d400fd40 con 0x7f35e8102fe0 2024-05-03T17:39:17.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.998+0000 7f35e5ffb700 1 -- 172.21.15.67:0/2844888444 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d400e3b0 con 0x7f35e8102fe0 2024-05-03T17:39:17.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.998+0000 7f35eead8700 1 -- 172.21.15.67:0/2844888444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35e81a8370 con 0x7f35e8102fe0 2024-05-03T17:39:17.000 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.998+0000 7f35eead8700 1 -- 172.21.15.67:0/2844888444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35e81a8860 con 0x7f35e8102fe0 2024-05-03T17:39:17.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:16.999+0000 7f35eead8700 1 -- 172.21.15.67:0/2844888444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35e806a190 con 0x7f35e8102fe0 2024-05-03T17:39:17.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:17.000+0000 7f35e5ffb700 1 -- 172.21.15.67:0/2844888444 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f35d4010610 con 0x7f35e8102fe0 2024-05-03T17:39:17.001 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:17.001+0000 7f35e5ffb700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f35d80b2990 0x7f35d80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:17.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:17.001+0000 7f35ec874700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f35d80b2990 0x7f35d80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:17.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:17.001+0000 7f35e5ffb700 1 -- 172.21.15.67:0/2844888444 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8506+0+0 (secure 0 0 0) 0x7f35d40e1e10 con 0x7f35e8102fe0 2024-05-03T17:39:17.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:17.001+0000 7f35ec874700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f35d80b2990 0x7f35d80b4e50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f35dc005950 tx=0x7f35dc00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:17.002 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:17.005+0000 7f35e5ffb700 1 -- 172.21.15.67:0/2844888444 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f35d40a98c0 con 0x7f35e8102fe0 2024-05-03T17:39:17.219 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:17.222+0000 7f35eead8700 1 -- 172.21.15.67:0/2844888444 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f35e802e750 con 0x7f35e8102fe0 2024-05-03T17:39:18.164 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.167+0000 7f35e5ffb700 1 -- 172.21.15.67:0/2844888444 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v83) ==== 181+0+0 (secure 0 0 0) 0x7f35d40a91c0 con 0x7f35e8102fe0 2024-05-03T17:39:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.172+0000 7f35eead8700 1 -- 172.21.15.67:0/2844888444 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f35d80b2990 msgr2=0x7f35d80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:18.168 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.172+0000 7f35eead8700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f35d80b2990 0x7f35d80b4e50 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f35dc005950 tx=0x7f35dc00a300 comp rx=0 tx=0).stop 2024-05-03T17:39:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.172+0000 7f35eead8700 1 -- 172.21.15.67:0/2844888444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e8102fe0 msgr2=0x7f35e81045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.172+0000 7f35eead8700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e8102fe0 0x7f35e81045c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f35d400d980 tx=0x7f35d400dc90 comp rx=0 tx=0).stop 2024-05-03T17:39:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.172+0000 7f35eead8700 1 -- 172.21.15.67:0/2844888444 shutdown_connections 2024-05-03T17:39:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.172+0000 7f35eead8700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f35d80b2990 0x7f35d80b4e50 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:18.169 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.172+0000 7f35eead8700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f35e81076c0 0x7f35e8104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:18.170 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.172+0000 7f35eead8700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f35e8102630 0x7f35e8104080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:18.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.172+0000 7f35eead8700 1 --2- 172.21.15.67:0/2844888444 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f35e8102fe0 0x7f35e81045c0 secure :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f35d400d980 tx=0x7f35d400dc90 comp rx=0 tx=0).stop 2024-05-03T17:39:18.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.172+0000 7f35eead8700 1 -- 172.21.15.67:0/2844888444 >> 172.21.15.67:0/2844888444 conn(0x7f35e8079350 msgr2=0x7f35e806bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:18.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.172+0000 7f35eead8700 1 -- 172.21.15.67:0/2844888444 shutdown_connections 2024-05-03T17:39:18.172 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:18.173+0000 7f35eead8700 1 -- 172.21.15.67:0/2844888444 wait complete. 2024-05-03T17:39:18.173 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' removed 2024-05-03T17:39:18.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:18 smithi067 ceph-mon[29731]: pgmap v857: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-05-03T17:39:18.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:18 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2844888444' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-03T17:39:18.510 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:18 smithi082 ceph-mon[33535]: pgmap v857: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-05-03T17:39:18.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:18 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2844888444' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-03T17:39:18.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:18 smithi070 ceph-mon[33250]: pgmap v857: 97 pgs: 97 active+clean; 638 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 4 op/s 2024-05-03T17:39:18.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:18 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2844888444' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-03T17:39:18.857 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea shell --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-03T17:39:19.119 INFO:teuthology.orchestra.run.smithi067.stderr:Inferring config /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/mon.a/config 2024-05-03T17:39:19.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:19 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/2844888444' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-03T17:39:19.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:19 smithi067 ceph-mon[29731]: osdmap e83: 12 total, 12 up, 12 in 2024-05-03T17:39:19.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:19 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/2844888444' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-03T17:39:19.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:19 smithi082 ceph-mon[33535]: osdmap e83: 12 total, 12 up, 12 in 2024-05-03T17:39:19.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:19 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/2844888444' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-03T17:39:19.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:19 smithi070 ceph-mon[33250]: osdmap e83: 12 total, 12 up, 12 in 2024-05-03T17:39:20.306 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:20 smithi067 ceph-mon[29731]: pgmap v859: 65 pgs: 65 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:39:20.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:20 smithi082 ceph-mon[33535]: pgmap v859: 65 pgs: 65 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:39:20.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:20 smithi070 ceph-mon[33250]: pgmap v859: 65 pgs: 65 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:39:20.928 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.931+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/3003554871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e48107700 msgr2=0x7f1e4810f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.931+0000 7f1e4e5c5700 1 --2- 172.21.15.67:0/3003554871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e48107700 0x7f1e4810f2d0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1e4000b0d0 tx=0x7f1e4000b3e0 comp rx=0 tx=0).stop 2024-05-03T17:39:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.931+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/3003554871 shutdown_connections 2024-05-03T17:39:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.931+0000 7f1e4e5c5700 1 --2- 172.21.15.67:0/3003554871 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e48107700 0x7f1e4810f2d0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.931+0000 7f1e4e5c5700 1 --2- 172.21.15.67:0/3003554871 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e48102620 0x7f1e48102a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.931+0000 7f1e4e5c5700 1 --2- 172.21.15.67:0/3003554871 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e48102fd0 0x7f1e48107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.931+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/3003554871 >> 172.21.15.67:0/3003554871 conn(0x7f1e48079440 msgr2=0x7f1e48079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:20.929 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.931+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/3003554871 shutdown_connections 2024-05-03T17:39:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.932+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/3003554871 wait complete. 2024-05-03T17:39:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.932+0000 7f1e4e5c5700 1 Processor -- start 2024-05-03T17:39:20.930 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.932+0000 7f1e4e5c5700 1 -- start start 2024-05-03T17:39:20.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.932+0000 7f1e4e5c5700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e48102620 0x7f1e481040c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:20.931 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.932+0000 7f1e4e5c5700 1 --2- >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e48102fd0 0x7f1e48104600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:20.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.932+0000 7f1e4e5c5700 1 --2- >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e48107700 0x7f1e48104b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:20.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.932+0000 7f1e4e5c5700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 -- 0x7f1e481125a0 con 0x7f1e48102620 2024-05-03T17:39:20.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.932+0000 7f1e4e5c5700 1 -- --> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] -- mon_getmap magic: 0 -- 0x7f1e48112420 con 0x7f1e48102fd0 2024-05-03T17:39:20.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.932+0000 7f1e4e5c5700 1 -- --> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] -- mon_getmap magic: 0 -- 0x7f1e48112720 con 0x7f1e48107700 2024-05-03T17:39:20.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e47fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e48102620 0x7f1e481040c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:20.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e47fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e48102620 0x7f1e481040c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:45862/0 (socket says 172.21.15.67:45862) 2024-05-03T17:39:20.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e47fff700 1 -- 172.21.15.67:0/1644114549 learned_addr learned my addr 172.21.15.67:0/1644114549 (peer_addr_for_me v2:172.21.15.67:0/0) 2024-05-03T17:39:20.932 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e477fe700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e48102fd0 0x7f1e48104600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:20.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e4cb62700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e48107700 0x7f1e48104b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:20.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e47fff700 1 -- 172.21.15.67:0/1644114549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e48107700 msgr2=0x7f1e48104b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:20.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e47fff700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e48107700 0x7f1e48104b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:20.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e47fff700 1 -- 172.21.15.67:0/1644114549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e48102fd0 msgr2=0x7f1e48104600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:20.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e47fff700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e48102fd0 0x7f1e48104600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:20.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e47fff700 1 -- 172.21.15.67:0/1644114549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e481a80b0 con 0x7f1e48102620 2024-05-03T17:39:20.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e4cb62700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e48107700 0x7f1e48104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-03T17:39:20.933 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e477fe700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e48102fd0 0x7f1e48104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-03T17:39:20.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e47fff700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e48102620 0x7f1e481040c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800d950 tx=0x7f1e3800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:20.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e457fa700 1 -- 172.21.15.67:0/1644114549 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e38016070 con 0x7f1e48102620 2024-05-03T17:39:20.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/1644114549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e481a83a0 con 0x7f1e48102620 2024-05-03T17:39:20.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e457fa700 1 -- 172.21.15.67:0/1644114549 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e38010460 con 0x7f1e48102620 2024-05-03T17:39:20.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e457fa700 1 -- 172.21.15.67:0/1644114549 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e3800f620 con 0x7f1e48102620 2024-05-03T17:39:20.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.933+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/1644114549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e481a8930 con 0x7f1e48102620 2024-05-03T17:39:20.934 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.934+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/1644114549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e4806a190 con 0x7f1e48102620 2024-05-03T17:39:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.936+0000 7f1e457fa700 1 -- 172.21.15.67:0/1644114549 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 17) ==== 149703+0+0 (secure 0 0 0) 0x7f1e38019070 con 0x7f1e48102620 2024-05-03T17:39:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.937+0000 7f1e457fa700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1e340b2910 0x7f1e340b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-03T17:39:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.937+0000 7f1e477fe700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1e340b2910 0x7f1e340b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-03T17:39:20.935 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.937+0000 7f1e457fa700 1 -- 172.21.15.67:0/1644114549 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7973+0+0 (secure 0 0 0) 0x7f1e380e1f60 con 0x7f1e48102620 2024-05-03T17:39:20.936 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.937+0000 7f1e477fe700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1e340b2910 0x7f1e340b4dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1e4806ccb0 tx=0x7f1e30005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-03T17:39:20.937 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:20.941+0000 7f1e457fa700 1 -- 172.21.15.67:0/1644114549 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191839 (secure 0 0 0) 0x7f1e380a9bd0 con 0x7f1e48102620 2024-05-03T17:39:21.154 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.157+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/1644114549 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1e481a8c50 con 0x7f1e48102620 2024-05-03T17:39:21.177 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.180+0000 7f1e457fa700 1 -- 172.21.15.67:0/1644114549 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v84) ==== 169+0+0 (secure 0 0 0) 0x7f1e380a94d0 con 0x7f1e48102620 2024-05-03T17:39:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.183+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/1644114549 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1e340b2910 msgr2=0x7f1e340b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.183+0000 7f1e4e5c5700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1e340b2910 0x7f1e340b4dd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1e4806ccb0 tx=0x7f1e30005d80 comp rx=0 tx=0).stop 2024-05-03T17:39:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.183+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/1644114549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e48102620 msgr2=0x7f1e481040c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-03T17:39:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.183+0000 7f1e4e5c5700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e48102620 0x7f1e481040c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1e3800d950 tx=0x7f1e3800dd10 comp rx=0 tx=0).stop 2024-05-03T17:39:21.181 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.184+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/1644114549 shutdown_connections 2024-05-03T17:39:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.184+0000 7f1e4e5c5700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.67:6800/3284262914,v1:172.21.15.67:6801/3284262914] conn(0x7f1e340b2910 0x7f1e340b4dd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.184+0000 7f1e4e5c5700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.82:3300/0,v1:172.21.15.82:6789/0] conn(0x7f1e48107700 0x7f1e48104b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.184+0000 7f1e4e5c5700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1e48102620 0x7f1e481040c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.184+0000 7f1e4e5c5700 1 --2- 172.21.15.67:0/1644114549 >> [v2:172.21.15.70:3300/0,v1:172.21.15.70:6789/0] conn(0x7f1e48102fd0 0x7f1e48104600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-03T17:39:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.184+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/1644114549 >> 172.21.15.67:0/1644114549 conn(0x7f1e48079440 msgr2=0x7f1e481062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-03T17:39:21.182 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.184+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/1644114549 shutdown_connections 2024-05-03T17:39:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:2024-05-03T17:39:21.184+0000 7f1e4e5c5700 1 -- 172.21.15.67:0/1644114549 wait complete. 2024-05-03T17:39:21.183 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' removed 2024-05-03T17:39:21.449 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:21 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1644114549' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-03T17:39:21.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:21 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1644114549' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-03T17:39:21.554 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:21 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1644114549' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-03T17:39:21.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T17:39:22.024 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T17:39:22.056 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T17:39:22.088 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-03T17:39:22.088 DEBUG:teuthology.orchestra.run.smithi067:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-03T17:39:22.106 DEBUG:teuthology.orchestra.run.smithi070:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-03T17:39:22.123 DEBUG:teuthology.orchestra.run.smithi082:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-03T17:39:22.146 INFO:tasks.cephadm:Stopping all daemons... 2024-05-03T17:39:22.146 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-03T17:39:22.146 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.a 2024-05-03T17:39:22.304 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:22 smithi070 ceph-mon[33250]: pgmap v860: 65 pgs: 65 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:39:22.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:22 smithi070 ceph-mon[33250]: from='client.? 172.21.15.67:0/1644114549' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-03T17:39:22.305 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:22 smithi070 ceph-mon[33250]: osdmap e84: 12 total, 12 up, 12 in 2024-05-03T17:39:22.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:22 smithi067 ceph-mon[29731]: pgmap v860: 65 pgs: 65 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:39:22.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:22 smithi067 ceph-mon[29731]: from='client.? 172.21.15.67:0/1644114549' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-03T17:39:22.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:22 smithi067 ceph-mon[29731]: osdmap e84: 12 total, 12 up, 12 in 2024-05-03T17:39:22.450 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:22 smithi067 systemd[1]: Stopping Ceph mon.a for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:39:22.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:22 smithi082 ceph-mon[33535]: pgmap v860: 65 pgs: 65 active+clean; 577 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-03T17:39:22.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:22 smithi082 ceph-mon[33535]: from='client.? 172.21.15.67:0/1644114549' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-03T17:39:22.511 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:22 smithi082 ceph-mon[33535]: osdmap e84: 12 total, 12 up, 12 in 2024-05-03T17:39:22.949 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:22 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a[29727]: 2024-05-03T17:39:22.464+0000 7f820ab66700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:39:22.950 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:22 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a[29727]: 2024-05-03T17:39:22.464+0000 7f820ab66700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-03T17:39:22.950 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:22 smithi067 podman[90588]: 2024-05-03 17:39:22.680519337 +0000 UTC m=+0.328127949 container died 60217e9ce4fcdb6735b0435e89a0320b40ce4733d201df85abad1c2f6054fd72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-05-03T17:39:23.338 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:23 smithi067 podman[90588]: 2024-05-03 17:39:23.074217362 +0000 UTC m=+0.721826016 container cleanup 60217e9ce4fcdb6735b0435e89a0320b40ce4733d201df85abad1c2f6054fd72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True) 2024-05-03T17:39:23.339 INFO:journalctl@ceph.mon.a.smithi067.stdout:May 03 17:39:23 smithi067 bash[90588]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-a 2024-05-03T17:39:23.498 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.a.service' 2024-05-03T17:39:23.540 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:39:23.541 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-03T17:39:23.541 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-03T17:39:23.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.b 2024-05-03T17:39:23.850 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:23 smithi070 systemd[1]: Stopping Ceph mon.b for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:39:24.117 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:23 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-b[33246]: 2024-05-03T17:39:23.852+0000 7f818e80d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:39:24.117 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:23 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-b[33246]: 2024-05-03T17:39:23.852+0000 7f818e80d700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-03T17:39:24.118 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:23 smithi070 podman[68417]: 2024-05-03 17:39:23.88404423 +0000 UTC m=+0.148133780 container died e15cdb79ea60485ba42dff319d4695b8d3baa19b79654df16fe7d7576253d57b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:39:24.502 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:24 smithi070 podman[68417]: 2024-05-03 17:39:24.279099051 +0000 UTC m=+0.543188600 container cleanup e15cdb79ea60485ba42dff319d4695b8d3baa19b79654df16fe7d7576253d57b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-03T17:39:24.502 INFO:journalctl@ceph.mon.b.smithi070.stdout:May 03 17:39:24 smithi070 bash[68417]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-b 2024-05-03T17:39:24.646 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.b.service' 2024-05-03T17:39:24.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:39:24.688 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-03T17:39:24.688 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-03T17:39:24.689 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.c 2024-05-03T17:39:24.999 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:24 smithi082 systemd[1]: Stopping Ceph mon.c for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:39:25.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-c[33531]: 2024-05-03T17:39:25.002+0000 7f9742b91700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:39:25.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-c[33531]: 2024-05-03T17:39:25.002+0000 7f9742b91700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-03T17:39:25.261 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:25 smithi082 podman[69593]: 2024-05-03 17:39:25.022958667 +0000 UTC m=+0.140026768 container died 2c1215b5a58674e908f6e47886ac0e96c3d81e5c0818f32b3f445e1d1a87684b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:39:25.761 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:25 smithi082 podman[69593]: 2024-05-03 17:39:25.553630334 +0000 UTC m=+0.670698461 container cleanup 2c1215b5a58674e908f6e47886ac0e96c3d81e5c0818f32b3f445e1d1a87684b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-c, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T17:39:25.761 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:25 smithi082 bash[69593]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-c 2024-05-03T17:39:26.013 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.c.service' 2024-05-03T17:39:26.034 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:25 smithi082 podman[69607]: 2024-05-03 17:39:25.780599767 +0000 UTC m=+0.758779268 container remove 2c1215b5a58674e908f6e47886ac0e96c3d81e5c0818f32b3f445e1d1a87684b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mon-c, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430) 2024-05-03T17:39:26.034 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:26 smithi082 systemd[1]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.c.service: Deactivated successfully. 2024-05-03T17:39:26.034 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:26 smithi082 systemd[1]: Stopped Ceph mon.c for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:39:26.034 INFO:journalctl@ceph.mon.c.smithi082.stdout:May 03 17:39:26 smithi082 systemd[1]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mon.c.service: Consumed 13.489s CPU time. 2024-05-03T17:39:26.072 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:39:26.073 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-03T17:39:26.073 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-03T17:39:26.073 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.x 2024-05-03T17:39:26.417 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:39:26 smithi067 systemd[1]: Stopping Ceph mgr.x for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:39:26.699 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:39:26 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x[29962]: 2024-05-03T17:39:26.420+0000 7f83b2fd3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:39:26.700 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:39:26 smithi067 podman[90710]: 2024-05-03 17:39:26.452598829 +0000 UTC m=+0.174953759 container died 0a733d8f609c66daba9bbd65dfafaac05d7969582a420d487ff5c1d8e7a77c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:39:27.136 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:39:26 smithi067 podman[90710]: 2024-05-03 17:39:26.846451007 +0000 UTC m=+0.568805933 container cleanup 0a733d8f609c66daba9bbd65dfafaac05d7969582a420d487ff5c1d8e7a77c97 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a047fe7599a549042d06a3fb5b6813a203eabea, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2) 2024-05-03T17:39:27.136 INFO:journalctl@ceph.mgr.x.smithi067.stdout:May 03 17:39:26 smithi067 bash[90710]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-x 2024-05-03T17:39:27.268 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.x.service' 2024-05-03T17:39:27.302 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:39:27.302 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-03T17:39:27.303 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-03T17:39:27.303 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.y 2024-05-03T17:39:27.693 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:39:27 smithi070 systemd[1]: Stopping Ceph mgr.y for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:39:27.947 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:39:27 smithi070 podman[68543]: 2024-05-03 17:39:27.69687223 +0000 UTC m=+0.187284757 container died c09dfad8badb707ab43c3a39a389971e9069127a7880d7165d5a19015625f131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0) 2024-05-03T17:39:28.240 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:39:28 smithi070 podman[68543]: 2024-05-03 17:39:28.034252332 +0000 UTC m=+0.524664854 container cleanup c09dfad8badb707ab43c3a39a389971e9069127a7880d7165d5a19015625f131 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-03T17:39:28.240 INFO:journalctl@ceph.mgr.y.smithi070.stdout:May 03 17:39:28 smithi070 bash[68543]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-y 2024-05-03T17:39:28.454 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.y.service' 2024-05-03T17:39:28.486 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:39:28.487 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-03T17:39:28.487 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-03T17:39:28.487 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.z 2024-05-03T17:39:28.817 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:39:28 smithi082 systemd[1]: Stopping Ceph mgr.z for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:39:29.124 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:39:28 smithi082 podman[69717]: 2024-05-03 17:39:28.821282496 +0000 UTC m=+0.142969136 container died d4f248973a3ef498632a528ef85f6d7fc0b27cbbcfa65efa34776d23824ad469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:39:29.482 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:39:29 smithi082 podman[69717]: 2024-05-03 17:39:29.226791811 +0000 UTC m=+0.548478451 container cleanup d4f248973a3ef498632a528ef85f6d7fc0b27cbbcfa65efa34776d23824ad469 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:39:29.483 INFO:journalctl@ceph.mgr.z.smithi082.stdout:May 03 17:39:29 smithi082 bash[69717]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mgr-z 2024-05-03T17:39:29.640 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mgr.z.service' 2024-05-03T17:39:29.672 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:39:29.672 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-03T17:39:29.672 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-03T17:39:29.673 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.0 2024-05-03T17:39:29.998 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:29 smithi067 systemd[1]: Stopping Ceph osd.0 for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:39:30.273 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:30 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0[44787]: 2024-05-03T17:39:30.001+0000 7f46f96eb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:39:30.273 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:30 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0[44787]: 2024-05-03T17:39:30.001+0000 7f46f96eb700 -1 osd.0 84 *** Got signal Terminated *** 2024-05-03T17:39:30.273 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:30 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0[44787]: 2024-05-03T17:39:30.001+0000 7f46f96eb700 -1 osd.0 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T17:39:35.841 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:35 smithi067 podman[90833]: 2024-05-03 17:39:35.533195134 +0000 UTC m=+5.655372270 container died 225b00db2e1fa5beecab080e392517738d33aa07e3ffbdd6252aebc3c3d2edc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-03T17:39:36.199 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:35 smithi067 podman[90833]: 2024-05-03 17:39:35.944017965 +0000 UTC m=+6.066195100 container cleanup 225b00db2e1fa5beecab080e392517738d33aa07e3ffbdd6252aebc3c3d2edc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:39:36.200 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:35 smithi067 bash[90833]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0 2024-05-03T17:39:36.480 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:36 smithi067 podman[90862]: 2024-05-03 17:39:36.230415417 +0000 UTC m=+0.688313171 container remove 225b00db2e1fa5beecab080e392517738d33aa07e3ffbdd6252aebc3c3d2edc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430) 2024-05-03T17:39:36.731 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:36 smithi067 podman[90954]: 2024-05-03 17:39:36.475257108 +0000 UTC m=+0.018877515 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:39:36.731 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:36 smithi067 podman[90954]: 2024-05-03 17:39:36.605876617 +0000 UTC m=+0.149497008 container create 0fa5c714b36ade44cc8663f92b56c1515de71be5c70f7534f5992964cf3ab1ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-05-03T17:39:37.174 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:36 smithi067 podman[90954]: 2024-05-03 17:39:36.866216845 +0000 UTC m=+0.409837236 container init 0fa5c714b36ade44cc8663f92b56c1515de71be5c70f7534f5992964cf3ab1ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-03T17:39:37.174 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:36 smithi067 podman[90954]: 2024-05-03 17:39:36.869597201 +0000 UTC m=+0.413217595 container start 0fa5c714b36ade44cc8663f92b56c1515de71be5c70f7534f5992964cf3ab1ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T17:39:37.174 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:36 smithi067 podman[90954]: 2024-05-03 17:39:36.966234644 +0000 UTC m=+0.509855068 container attach 0fa5c714b36ade44cc8663f92b56c1515de71be5c70f7534f5992964cf3ab1ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0) 2024-05-03T17:39:37.449 INFO:journalctl@ceph.osd.0.smithi067.stdout:May 03 17:39:37 smithi067 podman[90954]: 2024-05-03 17:39:37.177873299 +0000 UTC m=+0.721493693 container died 0fa5c714b36ade44cc8663f92b56c1515de71be5c70f7534f5992964cf3ab1ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:39:37.821 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.0.service' 2024-05-03T17:39:37.866 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:39:37.866 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-03T17:39:37.867 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-03T17:39:37.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.1 2024-05-03T17:39:38.151 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:37 smithi070 systemd[1]: Stopping Ceph osd.1 for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:39:38.554 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:38 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1[37861]: 2024-05-03T17:39:38.152+0000 7f87109e4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:39:38.555 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:38 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1[37861]: 2024-05-03T17:39:38.152+0000 7f87109e4700 -1 osd.1 84 *** Got signal Terminated *** 2024-05-03T17:39:38.555 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:38 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1[37861]: 2024-05-03T17:39:38.152+0000 7f87109e4700 -1 osd.1 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T17:39:43.784 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:43 smithi070 podman[68682]: 2024-05-03 17:39:43.51233342 +0000 UTC m=+5.459176062 container died 228df925f764f468ef733130138c3892a100d0bb25cd81148e65955aa11720c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5) 2024-05-03T17:39:44.054 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:43 smithi070 podman[68682]: 2024-05-03 17:39:43.871043481 +0000 UTC m=+5.817886132 container cleanup 228df925f764f468ef733130138c3892a100d0bb25cd81148e65955aa11720c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-05-03T17:39:44.054 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:43 smithi070 bash[68682]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1 2024-05-03T17:39:44.478 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:44 smithi070 podman[68699]: 2024-05-03 17:39:44.16409354 +0000 UTC m=+0.647708962 container remove 228df925f764f468ef733130138c3892a100d0bb25cd81148e65955aa11720c5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:39:44.804 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:44 smithi070 podman[68790]: 2024-05-03 17:39:44.481514302 +0000 UTC m=+0.103564520 container create c1a8629026a384c781e9aa5dcaeef5313a1b59b300e694289cbaa758bc5ebe4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:39:44.805 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:44 smithi070 podman[68790]: 2024-05-03 17:39:44.396117057 +0000 UTC m=+0.018167257 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:39:44.805 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:44 smithi070 podman[68790]: 2024-05-03 17:39:44.725711927 +0000 UTC m=+0.347762104 container init c1a8629026a384c781e9aa5dcaeef5313a1b59b300e694289cbaa758bc5ebe4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-03T17:39:44.805 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:44 smithi070 podman[68790]: 2024-05-03 17:39:44.728662662 +0000 UTC m=+0.350712835 container start c1a8629026a384c781e9aa5dcaeef5313a1b59b300e694289cbaa758bc5ebe4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:39:45.304 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:44 smithi070 podman[68790]: 2024-05-03 17:39:44.827975526 +0000 UTC m=+0.450025699 container attach c1a8629026a384c781e9aa5dcaeef5313a1b59b300e694289cbaa758bc5ebe4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0) 2024-05-03T17:39:45.305 INFO:journalctl@ceph.osd.1.smithi070.stdout:May 03 17:39:45 smithi070 podman[68790]: 2024-05-03 17:39:45.021590271 +0000 UTC m=+0.643640449 container died c1a8629026a384c781e9aa5dcaeef5313a1b59b300e694289cbaa758bc5ebe4c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430) 2024-05-03T17:39:45.625 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.1.service' 2024-05-03T17:39:45.662 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:39:45.663 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-03T17:39:45.663 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-03T17:39:45.663 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.2 2024-05-03T17:39:45.977 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:45 smithi082 systemd[1]: Stopping Ceph osd.2 for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:39:46.260 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:45 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2[38836]: 2024-05-03T17:39:45.980+0000 7fed54f35700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:39:46.261 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:45 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2[38836]: 2024-05-03T17:39:45.980+0000 7fed54f35700 -1 osd.2 84 *** Got signal Terminated *** 2024-05-03T17:39:46.261 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:45 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2[38836]: 2024-05-03T17:39:45.980+0000 7fed54f35700 -1 osd.2 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T17:39:51.694 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:51 smithi082 podman[69856]: 2024-05-03 17:39:51.386856951 +0000 UTC m=+5.531339884 container died e284ff54fd72fcb0bbd58f94fd0e579ee76256f2df0fdb35e379115e57549629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:39:52.011 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:51 smithi082 podman[69856]: 2024-05-03 17:39:51.788664951 +0000 UTC m=+5.933147883 container cleanup e284ff54fd72fcb0bbd58f94fd0e579ee76256f2df0fdb35e379115e57549629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2) 2024-05-03T17:39:52.011 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:51 smithi082 bash[69856]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2 2024-05-03T17:39:52.396 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:52 smithi082 podman[69876]: 2024-05-03 17:39:52.057095683 +0000 UTC m=+0.661983411 container remove e284ff54fd72fcb0bbd58f94fd0e579ee76256f2df0fdb35e379115e57549629 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:39:52.665 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:52 smithi082 podman[69965]: 2024-05-03 17:39:52.302677387 +0000 UTC m=+0.017843673 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:39:52.666 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:52 smithi082 podman[69965]: 2024-05-03 17:39:52.407961788 +0000 UTC m=+0.123128100 container create 5bae4e4681fca7b4a4c6e1de3f39de48a486cca3aa432f1ca4017a4ed742f39d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:39:52.666 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:52 smithi082 podman[69965]: 2024-05-03 17:39:52.669828953 +0000 UTC m=+0.384995223 container init 5bae4e4681fca7b4a4c6e1de3f39de48a486cca3aa432f1ca4017a4ed742f39d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=) 2024-05-03T17:39:52.963 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:52 smithi082 podman[69965]: 2024-05-03 17:39:52.673060919 +0000 UTC m=+0.388227201 container start 5bae4e4681fca7b4a4c6e1de3f39de48a486cca3aa432f1ca4017a4ed742f39d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True) 2024-05-03T17:39:52.964 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:52 smithi082 podman[69965]: 2024-05-03 17:39:52.794143527 +0000 UTC m=+0.509309808 container attach 5bae4e4681fca7b4a4c6e1de3f39de48a486cca3aa432f1ca4017a4ed742f39d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:39:53.260 INFO:journalctl@ceph.osd.2.smithi082.stdout:May 03 17:39:52 smithi082 podman[69965]: 2024-05-03 17:39:52.967788996 +0000 UTC m=+0.682955276 container died 5bae4e4681fca7b4a4c6e1de3f39de48a486cca3aa432f1ca4017a4ed742f39d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-03T17:39:53.647 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.2.service' 2024-05-03T17:39:53.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:39:53.688 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-03T17:39:53.688 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-03T17:39:53.688 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.3 2024-05-03T17:39:54.012 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:39:53 smithi067 systemd[1]: Stopping Ceph osd.3 for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:39:54.449 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:39:54 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3[51618]: 2024-05-03T17:39:54.015+0000 7fe779947700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:39:54.449 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:39:54 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3[51618]: 2024-05-03T17:39:54.015+0000 7fe779947700 -1 osd.3 84 *** Got signal Terminated *** 2024-05-03T17:39:54.450 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:39:54 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3[51618]: 2024-05-03T17:39:54.015+0000 7fe779947700 -1 osd.3 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T17:39:56.699 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:39:56 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3[51618]: 2024-05-03T17:39:56.208+0000 7fe771f33700 -1 osd.3 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:30.648557+0000 front 2024-05-03T17:39:30.648601+0000 (oldest deadline 2024-05-03T17:39:55.948719+0000) 2024-05-03T17:39:57.260 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:39:56 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:39:56.878+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:39:57.699 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:39:57 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3[51618]: 2024-05-03T17:39:57.239+0000 7fe771f33700 -1 osd.3 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:30.648557+0000 front 2024-05-03T17:39:30.648601+0000 (oldest deadline 2024-05-03T17:39:55.948719+0000) 2024-05-03T17:39:58.261 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:39:58 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:39:58.092+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:39:58.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:39:57 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:39:57.911+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:39:58.699 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:39:58 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3[51618]: 2024-05-03T17:39:58.222+0000 7fe771f33700 -1 osd.3 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:30.648557+0000 front 2024-05-03T17:39:30.648601+0000 (oldest deadline 2024-05-03T17:39:55.948719+0000) 2024-05-03T17:39:59.026 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:39:58 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:39:58.731+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:39:59.260 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:39:59 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:39:59.063+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:39:59.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:39:58 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:39:58.949+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:39:59.359 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:39:59 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:39:59.029+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:39:59.679 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:39:59 smithi067 podman[91062]: 2024-05-03 17:39:59.363634292 +0000 UTC m=+5.477361749 container died 86ee5670899126064f5645545a56098b9d50c6dcfd1e950d0b9c13ad77b43f52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:39:59.804 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:39:59 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:39:59.376+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:39:59.949 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:39:59 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:39:59.759+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:39:59.950 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:39:59 smithi067 podman[91062]: 2024-05-03 17:39:59.782138503 +0000 UTC m=+5.895865954 container cleanup 86ee5670899126064f5645545a56098b9d50c6dcfd1e950d0b9c13ad77b43f52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:39:59.950 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:39:59 smithi067 bash[91062]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3 2024-05-03T17:40:00.261 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:00 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:00.023+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:00.262 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:39:59 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:39:59.981+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:00.449 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:40:00 smithi067 podman[91080]: 2024-05-03 17:40:00.033646011 +0000 UTC m=+0.671695497 container remove 86ee5670899126064f5645545a56098b9d50c6dcfd1e950d0b9c13ad77b43f52 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_CLEAN=True, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-03T17:40:00.450 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:00 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:00.047+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:00.758 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:40:00 smithi067 podman[91174]: 2024-05-03 17:40:00.370076371 +0000 UTC m=+0.018579824 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:40:00.758 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:40:00 smithi067 podman[91174]: 2024-05-03 17:40:00.501159487 +0000 UTC m=+0.149662917 container create bbe43cc71e65290ea214bbfbf2fc2972b61c583f7af84dcbba212cc51d8f6cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:40:00.804 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:00 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:00.350+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:01.056 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:40:00 smithi067 podman[91174]: 2024-05-03 17:40:00.762052925 +0000 UTC m=+0.410556355 container init bbe43cc71e65290ea214bbfbf2fc2972b61c583f7af84dcbba212cc51d8f6cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-03T17:40:01.056 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:40:00 smithi067 podman[91174]: 2024-05-03 17:40:00.765309465 +0000 UTC m=+0.413812896 container start bbe43cc71e65290ea214bbfbf2fc2972b61c583f7af84dcbba212cc51d8f6cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:40:01.056 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:40:00 smithi067 podman[91174]: 2024-05-03 17:40:00.870519191 +0000 UTC m=+0.519022621 container attach bbe43cc71e65290ea214bbfbf2fc2972b61c583f7af84dcbba212cc51d8f6cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T17:40:01.057 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:00 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:00.772+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:01.260 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:01 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:01.013+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:01.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:00 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:00.968+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:01.378 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:01 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:01.058+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:01.379 INFO:journalctl@ceph.osd.3.smithi067.stdout:May 03 17:40:01 smithi067 podman[91174]: 2024-05-03 17:40:01.071790789 +0000 UTC m=+0.720294220 container died bbe43cc71e65290ea214bbfbf2fc2972b61c583f7af84dcbba212cc51d8f6cb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:40:01.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:01 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:01.302+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:01.715 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.3.service' 2024-05-03T17:40:01.759 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:40:01.759 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-03T17:40:01.759 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-03T17:40:01.759 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.4 2024-05-03T17:40:01.949 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:01 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:01.778+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:02.054 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:01 smithi070 systemd[1]: Stopping Ceph osd.4 for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:40:02.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:01 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:01.988+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:02.261 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:02 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:02.042+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:02.449 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:02 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:02.015+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:02.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:02 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:02.296+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:02.554 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:02 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4[44798]: 2024-05-03T17:40:02.090+0000 7fb0da765700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:40:02.555 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:02 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4[44798]: 2024-05-03T17:40:02.090+0000 7fb0da765700 -1 osd.4 84 *** Got signal Terminated *** 2024-05-03T17:40:02.555 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:02 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4[44798]: 2024-05-03T17:40:02.090+0000 7fb0da765700 -1 osd.4 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T17:40:03.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:02 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:02.748+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:03.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:02 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:02.992+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:03.260 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:03 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:03.090+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:03.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:02 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:02.955+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:03.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:03 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:03.266+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:04.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:03 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:03.739+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:04.199 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:03 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:03.986+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:04.260 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:04 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:04.069+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:04.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:03 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:03.976+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:04.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:04 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:04.292+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:05.199 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:04 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:04.942+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:05.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:04 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:04.727+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:05.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:04 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:04.727+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:05.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:04 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:04.989+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:05.261 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:05 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:05.055+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:05.554 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:05 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4[44798]: 2024-05-03T17:40:05.237+0000 7fb0d6d59700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.264480+0000 front 2024-05-03T17:39:39.264508+0000 (oldest deadline 2024-05-03T17:40:04.564604+0000) 2024-05-03T17:40:05.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:05 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:05.322+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:05.977 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:05.708+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:05.977 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:05.708+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:06.261 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:06 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:06.101+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:06.261 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:06 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:06.101+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.422641+0000 front 2024-05-03T17:39:37.422653+0000 (oldest deadline 2024-05-03T17:40:05.623280+0000) 2024-05-03T17:40:06.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:05 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:05.975+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:06.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:05 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:05.975+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:06.450 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:05.981+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:06.455 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:06 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:06.115+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:06.455 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:06 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4[44798]: 2024-05-03T17:40:06.213+0000 7fb0d6d59700 -1 osd.4 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.264480+0000 front 2024-05-03T17:39:39.264508+0000 (oldest deadline 2024-05-03T17:40:04.564604+0000) 2024-05-03T17:40:06.455 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:06 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:06.350+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:06.456 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:06 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:06.350+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:07.199 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:06 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:06.932+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:07.199 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:06 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:06.932+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:07.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:06 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:06.711+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:07.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:06 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:06.711+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:07.260 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:07 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:07.084+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:07.260 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:07 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:07.084+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.422641+0000 front 2024-05-03T17:39:37.422653+0000 (oldest deadline 2024-05-03T17:40:05.623280+0000) 2024-05-03T17:40:07.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:07 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:07.003+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:07.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:07 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:07.003+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:07.384 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:07 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:07.127+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:07.723 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:07 smithi070 podman[68898]: 2024-05-03 17:40:07.421441608 +0000 UTC m=+5.455438520 container stop aee95c03016d51d29b8442ae8d438d72b40dbdaff194ac1e8bc5003a90281582 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-03T17:40:07.724 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:07 smithi070 podman[68898]: 2024-05-03 17:40:07.447061645 +0000 UTC m=+5.481058564 container died aee95c03016d51d29b8442ae8d438d72b40dbdaff194ac1e8bc5003a90281582 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:40:07.724 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:07 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:07.387+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:07.724 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:07 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:07.387+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:07.760 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:07 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:07.281+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:08.054 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:07 smithi070 podman[68898]: 2024-05-03 17:40:07.865559503 +0000 UTC m=+5.899556417 container cleanup aee95c03016d51d29b8442ae8d438d72b40dbdaff194ac1e8bc5003a90281582 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-05-03T17:40:08.054 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:07 smithi070 bash[68898]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4 2024-05-03T17:40:08.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:07 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:07.729+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:08.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:07 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:07.729+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:08.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:07 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:07.896+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:08.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:07 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:07.896+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:08.371 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:08 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:08.094+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:08.371 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:08 smithi070 podman[68915]: 2024-05-03 17:40:08.075263113 +0000 UTC m=+0.646921228 container remove aee95c03016d51d29b8442ae8d438d72b40dbdaff194ac1e8bc5003a90281582 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430) 2024-05-03T17:40:08.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:08 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:08.236+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:08.511 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:08 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:08.054+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:08.511 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:08 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:08.054+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.422641+0000 front 2024-05-03T17:39:37.422653+0000 (oldest deadline 2024-05-03T17:40:05.623280+0000) 2024-05-03T17:40:08.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:08 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:08.051+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:08.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:08 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:08.051+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:08.663 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:08 smithi070 podman[69017]: 2024-05-03 17:40:08.403315171 +0000 UTC m=+0.018233650 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:40:08.663 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:08 smithi070 podman[69017]: 2024-05-03 17:40:08.526245195 +0000 UTC m=+0.141163664 container create 4907b7f3b7395ceaf5fa00d4aa82b012a0623dd19345a20dbeb79ad0394d95ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:40:08.664 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:08 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:08.374+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:08.664 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:08 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:08.374+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:09.054 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:08 smithi070 podman[69017]: 2024-05-03 17:40:08.804148556 +0000 UTC m=+0.419067017 container init 4907b7f3b7395ceaf5fa00d4aa82b012a0623dd19345a20dbeb79ad0394d95ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4-deactivate, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430) 2024-05-03T17:40:09.054 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:08 smithi070 podman[69017]: 2024-05-03 17:40:08.807381262 +0000 UTC m=+0.422299724 container start 4907b7f3b7395ceaf5fa00d4aa82b012a0623dd19345a20dbeb79ad0394d95ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T17:40:09.054 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:08 smithi070 podman[69017]: 2024-05-03 17:40:08.903922168 +0000 UTC m=+0.518840641 container attach 4907b7f3b7395ceaf5fa00d4aa82b012a0623dd19345a20dbeb79ad0394d95ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-03T17:40:09.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:08 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:08.747+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:09.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:08 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:08.747+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:09.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:08 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:08.861+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:09.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:08 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:08.861+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:09.261 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:09 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:09.213+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:09.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:09 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:09.088+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:09.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:09 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:09.088+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:09.261 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:09 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:09.004+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:09.262 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:09 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:09.004+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.422641+0000 front 2024-05-03T17:39:37.422653+0000 (oldest deadline 2024-05-03T17:40:05.623280+0000) 2024-05-03T17:40:09.347 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:09 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:09.136+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:09.348 INFO:journalctl@ceph.osd.4.smithi070.stdout:May 03 17:40:09 smithi070 podman[69017]: 2024-05-03 17:40:09.095029838 +0000 UTC m=+0.709948302 container died 4907b7f3b7395ceaf5fa00d4aa82b012a0623dd19345a20dbeb79ad0394d95ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-03T17:40:09.704 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:09 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:09.350+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:09.705 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:09 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:09.350+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:09.736 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.4.service' 2024-05-03T17:40:09.775 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:40:09.776 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-03T17:40:09.776 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-03T17:40:09.776 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.5 2024-05-03T17:40:10.079 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:09 smithi082 systemd[1]: Stopping Ceph osd.5 for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:40:10.079 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:10.042+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:10.079 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:10.042+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.422641+0000 front 2024-05-03T17:39:37.422653+0000 (oldest deadline 2024-05-03T17:40:05.623280+0000) 2024-05-03T17:40:10.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:09 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:09.735+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:10.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:09 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:09.735+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:10.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:09 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:09.891+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:10.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:09 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:09.891+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:10.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:10.222+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:10.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:10.081+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:10.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:10.081+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:10.512 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:10.092+0000 7fcfb03f7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:40:10.512 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:10.092+0000 7fcfb03f7700 -1 osd.5 84 *** Got signal Terminated *** 2024-05-03T17:40:10.512 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:10 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:10.092+0000 7fcfb03f7700 -1 osd.5 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T17:40:10.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:10 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:10.169+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:10.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:10 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:10.323+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:10.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:10 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:10.323+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:11.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:10 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:10.747+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:11.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:10 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:10.747+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:11.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:10 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:10.914+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:11.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:10 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:10.914+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:11.510 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:11 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:11.232+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:11.511 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:11 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:11.024+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:11.511 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:11 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:11.024+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.422641+0000 front 2024-05-03T17:39:37.422653+0000 (oldest deadline 2024-05-03T17:40:05.623280+0000) 2024-05-03T17:40:11.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:11 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:11.041+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:11.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:11 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:11.041+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:11.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:11 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:11.147+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:11.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:11 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:11.288+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:11.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:11 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:11.288+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:12.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:11 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:11.787+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:12.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:11 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:11.787+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:12.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:11 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:11.888+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:12.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:11 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:11.888+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:12.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:11 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:11.888+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:49.562413+0000 front 2024-05-03T17:39:49.562365+0000 (oldest deadline 2024-05-03T17:40:11.862186+0000) 2024-05-03T17:40:12.261 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:11 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:11.979+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:12.261 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:11 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:11.979+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.422641+0000 front 2024-05-03T17:39:37.422653+0000 (oldest deadline 2024-05-03T17:40:05.623280+0000) 2024-05-03T17:40:12.261 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:11 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:11.979+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.623578+0000 front 2024-05-03T17:39:45.623645+0000 (oldest deadline 2024-05-03T17:40:11.523711+0000) 2024-05-03T17:40:12.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:12 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:12.073+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:12.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:12 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:12.073+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:12.262 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:12 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:12.073+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:12.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:12 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:12.239+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:12.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:12 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:12.239+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:12.555 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:12 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:12.156+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:12.555 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:12 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:12.156+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:12.760 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:12 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:12.273+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:13.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:12 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:12.738+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:13.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:12 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:12.738+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:13.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:12 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:12.860+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:13.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:12 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:12.860+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:13.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:12 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:12.860+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:49.562413+0000 front 2024-05-03T17:39:49.562365+0000 (oldest deadline 2024-05-03T17:40:11.862186+0000) 2024-05-03T17:40:13.243 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:12 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:12.978+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:13.243 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:12 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:12.978+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.422641+0000 front 2024-05-03T17:39:37.422653+0000 (oldest deadline 2024-05-03T17:40:05.623280+0000) 2024-05-03T17:40:13.244 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:12 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:12.978+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.623578+0000 front 2024-05-03T17:39:45.623645+0000 (oldest deadline 2024-05-03T17:40:11.523711+0000) 2024-05-03T17:40:13.244 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:13 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:13.057+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:13.244 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:13 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:13.057+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:13.244 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:13 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:13.057+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:13.510 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:13 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:13.247+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:13.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:13 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:13.247+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:13.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:13 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:13.234+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:13.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:13 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:13.234+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:13.555 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:13 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:13.129+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:13.555 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:13 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:13.129+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:14.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:13 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:13.768+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:14.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:13 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:13.768+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:14.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:13 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:13.768+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:14.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:13 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:13.846+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:14.201 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:13 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:13.846+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:14.201 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:13 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:13.846+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:49.562413+0000 front 2024-05-03T17:39:49.562365+0000 (oldest deadline 2024-05-03T17:40:11.862186+0000) 2024-05-03T17:40:14.257 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:14 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:14.098+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:14.257 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:14 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:14.098+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:14.257 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:14 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:14.098+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:14.257 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:13 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:13.977+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:14.258 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:13 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:13.977+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.422641+0000 front 2024-05-03T17:39:37.422653+0000 (oldest deadline 2024-05-03T17:40:05.623280+0000) 2024-05-03T17:40:14.258 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:13 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:13.977+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.623578+0000 front 2024-05-03T17:39:45.623645+0000 (oldest deadline 2024-05-03T17:40:11.523711+0000) 2024-05-03T17:40:14.510 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:14 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:14.260+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:14.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:14 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:14.260+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:14.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:14 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:14.117+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:14.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:14 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:14.117+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:14.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:14 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:14.252+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:14.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:14 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:14.252+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:14.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:14 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:14.252+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:15.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:14 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:14.786+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:15.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:14 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:14.786+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:15.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:14 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:14.786+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:15.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:14 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:14.890+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:15.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:14 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:14.890+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:15.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:14 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:14.890+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:49.562413+0000 front 2024-05-03T17:39:49.562365+0000 (oldest deadline 2024-05-03T17:40:11.862186+0000) 2024-05-03T17:40:15.239 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:14 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:14.966+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.722450+0000 front 2024-05-03T17:39:32.722400+0000 (oldest deadline 2024-05-03T17:39:57.422237+0000) 2024-05-03T17:40:15.239 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:14 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:14.966+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.422641+0000 front 2024-05-03T17:39:37.422653+0000 (oldest deadline 2024-05-03T17:40:05.623280+0000) 2024-05-03T17:40:15.239 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:14 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5[45649]: 2024-05-03T17:40:14.966+0000 7fcfac9eb700 -1 osd.5 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.623578+0000 front 2024-05-03T17:39:45.623645+0000 (oldest deadline 2024-05-03T17:40:11.523711+0000) 2024-05-03T17:40:15.240 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:15 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:15.077+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:15.240 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:15 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:15.077+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:15.240 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:15 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:15.077+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:15.504 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:15 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:15.242+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:15.504 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:15 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:15.242+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:15.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:15 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:15.124+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:15.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:15 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:15.124+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:15.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:15 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:15.265+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:15.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:15 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:15.265+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:15.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:15 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:15.265+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:15.760 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:15 smithi082 podman[70098]: 2024-05-03 17:40:15.507597538 +0000 UTC m=+5.531961322 container died e2bb673a94c0e7ad598d60bb443d4f4af6241d7f632953cfaa89854a20e372d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:40:16.159 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:15 smithi082 podman[70098]: 2024-05-03 17:40:15.902429616 +0000 UTC m=+5.926793392 container cleanup e2bb673a94c0e7ad598d60bb443d4f4af6241d7f632953cfaa89854a20e372d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T17:40:16.159 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:15 smithi082 bash[70098]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5 2024-05-03T17:40:16.159 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:16 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:16.047+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:16.159 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:16 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:16.047+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:16.159 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:16 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:16.047+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:16.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:15 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:15.738+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:16.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:15 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:15.738+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:16.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:15 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:15.738+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:16.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:15 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:15.928+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:16.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:15 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:15.928+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:16.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:15 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:15.928+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:49.562413+0000 front 2024-05-03T17:39:49.562365+0000 (oldest deadline 2024-05-03T17:40:11.862186+0000) 2024-05-03T17:40:16.511 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:16 smithi082 podman[70115]: 2024-05-03 17:40:16.162830431 +0000 UTC m=+0.649104334 container remove e2bb673a94c0e7ad598d60bb443d4f4af6241d7f632953cfaa89854a20e372d3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5, ceph=True, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:40:16.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:16 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:16.290+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:16.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:16 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:16.290+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:16.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:16 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:16.170+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:16.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:16 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:16.170+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:16.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:16 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:16.270+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:16.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:16 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:16.270+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:16.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:16 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:16.270+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:16.863 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:16 smithi082 podman[70210]: 2024-05-03 17:40:16.42399832 +0000 UTC m=+0.017696818 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:40:16.863 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:16 smithi082 podman[70210]: 2024-05-03 17:40:16.539523957 +0000 UTC m=+0.133222428 container create 724c71944bc4fa8544a05b5f1efe13770f26f82f2f8e9dd147b0dadc017c4911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5-deactivate, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:40:16.864 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:16 smithi082 podman[70210]: 2024-05-03 17:40:16.767202429 +0000 UTC m=+0.360900905 container init 724c71944bc4fa8544a05b5f1efe13770f26f82f2f8e9dd147b0dadc017c4911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:40:16.864 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:16 smithi082 podman[70210]: 2024-05-03 17:40:16.77026445 +0000 UTC m=+0.363962918 container start 724c71944bc4fa8544a05b5f1efe13770f26f82f2f8e9dd147b0dadc017c4911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-03T17:40:16.864 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:16 smithi082 podman[70210]: 2024-05-03 17:40:16.867295508 +0000 UTC m=+0.460993990 container attach 724c71944bc4fa8544a05b5f1efe13770f26f82f2f8e9dd147b0dadc017c4911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-03T17:40:17.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:16 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:16.777+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:17.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:16 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:16.777+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:17.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:16 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:16.777+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:17.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:16 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:16.969+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:17.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:16 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:16.969+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:17.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:16 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:16.969+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:49.562413+0000 front 2024-05-03T17:39:49.562365+0000 (oldest deadline 2024-05-03T17:40:11.862186+0000) 2024-05-03T17:40:17.259 INFO:journalctl@ceph.osd.5.smithi082.stdout:May 03 17:40:17 smithi082 podman[70210]: 2024-05-03 17:40:17.069566807 +0000 UTC m=+0.663265277 container died 724c71944bc4fa8544a05b5f1efe13770f26f82f2f8e9dd147b0dadc017c4911 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T17:40:17.260 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:17 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:17.086+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:17.260 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:17 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:17.086+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:17.260 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:17 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:17.086+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:17.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:17 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:17.263+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:17.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:17 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:17.263+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:17.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:17 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:17.133+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:17.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:17 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:17.133+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:17.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:17 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:17.252+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:17.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:17 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:17.252+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:17.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:17 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:17.252+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:17.716 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.5.service' 2024-05-03T17:40:17.754 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:40:17.754 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-03T17:40:17.754 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-03T17:40:17.754 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.6 2024-05-03T17:40:18.090 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:17 smithi067 systemd[1]: Stopping Ceph osd.6 for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:40:18.090 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:17 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:17.953+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:18.090 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:17 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:17.953+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:18.090 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:17 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:17.953+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:49.562413+0000 front 2024-05-03T17:39:49.562365+0000 (oldest deadline 2024-05-03T17:40:11.862186+0000) 2024-05-03T17:40:18.090 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:17 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:17.779+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:18.091 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:17 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:17.779+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:18.091 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:17 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:17.779+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:18.449 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:18 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:18.093+0000 7f3f237f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:40:18.449 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:18 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:18.093+0000 7f3f237f2700 -1 osd.6 84 *** Got signal Terminated *** 2024-05-03T17:40:18.450 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:18 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:18.093+0000 7f3f237f2700 -1 osd.6 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T17:40:18.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:18 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:18.298+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:18.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:18 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:18.298+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:18.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:18 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:18.130+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:18.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:18 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:18.130+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:18.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:18 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:18.130+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:18.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:18 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:18.171+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:18.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:18 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:18.171+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:18.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:18 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:18.233+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:18.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:18 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:18.233+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:18.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:18 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:18.233+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:19.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:18 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:18.795+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:19.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:18 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:18.795+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:19.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:18 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:18.795+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:19.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:18 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:18.938+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:19.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:18 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:18.938+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:19.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:18 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:18.938+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:49.562413+0000 front 2024-05-03T17:39:49.562365+0000 (oldest deadline 2024-05-03T17:40:11.862186+0000) 2024-05-03T17:40:19.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:19 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:19.257+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:19.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:19 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:19.257+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:19.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:19 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:19.171+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:19.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:19 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:19.171+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:19.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:19 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:19.171+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:19.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:19 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:19.221+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:19.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:19 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:19.221+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:19.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:19 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:19.221+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:19.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:19 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:19.162+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:19.555 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:19 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:19.162+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:20.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:19 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:19.797+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:20.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:19 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:19.797+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:20.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:19 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:19.797+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:20.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:19 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:19.967+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:20.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:19 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:19.967+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:20.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:19 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:19.967+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:49.562413+0000 front 2024-05-03T17:39:49.562365+0000 (oldest deadline 2024-05-03T17:40:11.862186+0000) 2024-05-03T17:40:20.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:19 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:19.967+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.163370+0000 front 2024-05-03T17:39:54.163351+0000 (oldest deadline 2024-05-03T17:40:19.463391+0000) 2024-05-03T17:40:20.510 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:20 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:20.263+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:20.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:20 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:20.263+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:20.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:20 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:20.124+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:20.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:20 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:20.124+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:20.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:20 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:20.124+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:20.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:20 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:20.170+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:20.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:20 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:20.170+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:20.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:20 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:20.170+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.538066+0000 front 2024-05-03T17:39:57.538215+0000 (oldest deadline 2024-05-03T17:40:19.838038+0000) 2024-05-03T17:40:20.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:20 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:20.235+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:20.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:20 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:20.235+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:20.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:20 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:20.235+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:21.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:20 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:20.830+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:21.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:20 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:20.830+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:21.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:20 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:20.830+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:21.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:20 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:20.954+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:21.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:20 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:20.954+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:21.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:20 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:20.954+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:49.562413+0000 front 2024-05-03T17:39:49.562365+0000 (oldest deadline 2024-05-03T17:40:11.862186+0000) 2024-05-03T17:40:21.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:20 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:20.954+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.163370+0000 front 2024-05-03T17:39:54.163351+0000 (oldest deadline 2024-05-03T17:40:19.463391+0000) 2024-05-03T17:40:21.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:21 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:21.106+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:21.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:21 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:21.106+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:21.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:21 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:21.106+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:21.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:21 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:21.279+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:21.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:21 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:21.279+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:21.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:21 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:21.279+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:21.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:21 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:21.203+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:21.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:21 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:21.203+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:21.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:21 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:21.203+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.538066+0000 front 2024-05-03T17:39:57.538215+0000 (oldest deadline 2024-05-03T17:40:19.838038+0000) 2024-05-03T17:40:21.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:21 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:21.195+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:21.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:21 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:21.195+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:21.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:21 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:21.195+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:21.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:21 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:21.195+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:22.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:21 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:21.806+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:22.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:21 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:21.806+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:22.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:21 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:21.806+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:22.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:21 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:21.960+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:22.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:21 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:21.960+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:22.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:21 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:21.960+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:49.562413+0000 front 2024-05-03T17:39:49.562365+0000 (oldest deadline 2024-05-03T17:40:11.862186+0000) 2024-05-03T17:40:22.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:21 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:21.960+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.163370+0000 front 2024-05-03T17:39:54.163351+0000 (oldest deadline 2024-05-03T17:40:19.463391+0000) 2024-05-03T17:40:22.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:22 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:22.296+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:22.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:22 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:22.296+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:22.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:22 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:22.296+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:22.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:22 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:22.144+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:22.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:22 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:22.144+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:22.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:22 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:22.144+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:22.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:22.222+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:22.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:22.222+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:22.555 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:22.222+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.538066+0000 front 2024-05-03T17:39:57.538215+0000 (oldest deadline 2024-05-03T17:40:19.838038+0000) 2024-05-03T17:40:22.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:22.156+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:22.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:22.156+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:22.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:22.156+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:22.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:22.156+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:23.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:22 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:22.788+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:23.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:22 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:22.788+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:23.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:22 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:22.788+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:23.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:23 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:23.010+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.158080+0000 front 2024-05-03T17:39:34.158317+0000 (oldest deadline 2024-05-03T17:39:58.858229+0000) 2024-05-03T17:40:23.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:23 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:23.010+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:41.560417+0000 front 2024-05-03T17:39:41.560214+0000 (oldest deadline 2024-05-03T17:40:06.260265+0000) 2024-05-03T17:40:23.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:23 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:23.010+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:49.562413+0000 front 2024-05-03T17:39:49.562365+0000 (oldest deadline 2024-05-03T17:40:11.862186+0000) 2024-05-03T17:40:23.200 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:23 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6[58593]: 2024-05-03T17:40:23.010+0000 7f3f1bdde700 -1 osd.6 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.163370+0000 front 2024-05-03T17:39:54.163351+0000 (oldest deadline 2024-05-03T17:40:19.463391+0000) 2024-05-03T17:40:23.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:23 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:23.128+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:23.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:23 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:23.128+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:23.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:23 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:23.128+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:23.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:23 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:23.248+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:23.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:23 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:23.248+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:23.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:23 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:23.248+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:23.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:23 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:23.177+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:23.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:23 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:23.177+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:23.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:23 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:23.177+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.538066+0000 front 2024-05-03T17:39:57.538215+0000 (oldest deadline 2024-05-03T17:40:19.838038+0000) 2024-05-03T17:40:23.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:23 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:23.203+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:23.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:23 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:23.203+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:23.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:23 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:23.203+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:23.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:23 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:23.203+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:23.915 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:23 smithi067 podman[91278]: 2024-05-03 17:40:23.631284005 +0000 UTC m=+5.673723709 container died d34d397dffd104116b430735f138b6192b18fa170bed82d624eee2ab7aaf36ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:40:23.916 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:23 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:23.739+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:23.916 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:23 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:23.739+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:23.916 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:23 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:23.739+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:23.916 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:23 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:23.739+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:24.199 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:24 smithi067 podman[91278]: 2024-05-03 17:40:24.026907683 +0000 UTC m=+6.069347396 container cleanup d34d397dffd104116b430735f138b6192b18fa170bed82d624eee2ab7aaf36ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-05-03T17:40:24.199 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:24 smithi067 bash[91278]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6 2024-05-03T17:40:24.510 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:24 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:24.252+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:24.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:24 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:24.252+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:24.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:24 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:24.252+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:24.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:24 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:24.113+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:24.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:24 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:24.113+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:24.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:24 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:24.113+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:24.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:24 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:24.141+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:24.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:24 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:24.141+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:24.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:24 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:24.141+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.538066+0000 front 2024-05-03T17:39:57.538215+0000 (oldest deadline 2024-05-03T17:40:19.838038+0000) 2024-05-03T17:40:24.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:24 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:24.163+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:24.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:24 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:24.163+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:24.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:24 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:24.163+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:24.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:24 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:24.163+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:24.650 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:24 smithi067 podman[91295]: 2024-05-03 17:40:24.303198014 +0000 UTC m=+0.670745524 container remove d34d397dffd104116b430735f138b6192b18fa170bed82d624eee2ab7aaf36ee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-03T17:40:24.944 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:24 smithi067 podman[91385]: 2024-05-03 17:40:24.557550519 +0000 UTC m=+0.018350496 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:40:24.945 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:24 smithi067 podman[91385]: 2024-05-03 17:40:24.670856025 +0000 UTC m=+0.131656039 container create 31688c02b2d9d4176432543b9f65f8956d2dd0d9e8c7c6aa84bac1fdbc4c0fee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-03T17:40:24.945 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:24 smithi067 podman[91385]: 2024-05-03 17:40:24.865088575 +0000 UTC m=+0.325888561 container init 31688c02b2d9d4176432543b9f65f8956d2dd0d9e8c7c6aa84bac1fdbc4c0fee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-05-03T17:40:24.945 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:24 smithi067 podman[91385]: 2024-05-03 17:40:24.868543958 +0000 UTC m=+0.329343930 container start 31688c02b2d9d4176432543b9f65f8956d2dd0d9e8c7c6aa84bac1fdbc4c0fee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD) 2024-05-03T17:40:24.945 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:24 smithi067 podman[91385]: 2024-05-03 17:40:24.948061852 +0000 UTC m=+0.408861825 container attach 31688c02b2d9d4176432543b9f65f8956d2dd0d9e8c7c6aa84bac1fdbc4c0fee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:40:24.945 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:24 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:24.766+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:24.945 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:24 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:24.766+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:24.946 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:24 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:24.766+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:24.946 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:24 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:24.766+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:25.199 INFO:journalctl@ceph.osd.6.smithi067.stdout:May 03 17:40:25 smithi067 podman[91385]: 2024-05-03 17:40:25.179850933 +0000 UTC m=+0.640650908 container died 31688c02b2d9d4176432543b9f65f8956d2dd0d9e8c7c6aa84bac1fdbc4c0fee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:40:25.510 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:25.218+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:25.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:25.218+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:25.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:25.218+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:25.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:25.113+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:25.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:25.113+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:25.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:25 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:25.113+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:25.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:25 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:25.119+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:25.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:25 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:25.119+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:25.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:25 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:25.119+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.538066+0000 front 2024-05-03T17:39:57.538215+0000 (oldest deadline 2024-05-03T17:40:19.838038+0000) 2024-05-03T17:40:25.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:25 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:25.161+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:25.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:25 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:25.161+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:25.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:25 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:25.161+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:25.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:25 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:25.161+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:25.831 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.6.service' 2024-05-03T17:40:25.870 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:40:25.870 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-03T17:40:25.870 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-03T17:40:25.870 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.7 2024-05-03T17:40:26.181 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:26.125+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:26.181 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:26.125+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:26.181 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:26.125+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:26.181 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:26.125+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:26.181 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:25 smithi070 systemd[1]: Stopping Ceph osd.7 for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:40:26.181 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:26.124+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:26.182 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:26.124+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:26.182 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:26.124+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.538066+0000 front 2024-05-03T17:39:57.538215+0000 (oldest deadline 2024-05-03T17:40:19.838038+0000) 2024-05-03T17:40:26.182 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:26.184+0000 7fd8b572d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:40:26.182 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:26.184+0000 7fd8b572d700 -1 osd.7 84 *** Got signal Terminated *** 2024-05-03T17:40:26.182 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:26 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:26.184+0000 7fd8b572d700 -1 osd.7 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T17:40:26.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:25 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:25.814+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:26.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:25 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:25.814+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:26.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:25 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:25.814+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:26.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:25 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:25.814+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:26.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:26 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:26.163+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:26.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:26 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:26.163+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:26.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:26 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:26.163+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:26.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:26 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:26.216+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:26.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:26 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:26.216+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:26.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:26 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:26.216+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:27.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:26 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:26.794+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:27.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:26 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:26.794+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:27.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:26 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:26.794+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:27.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:26 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:26.794+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:27.510 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:27 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:27.212+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:27.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:27 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:27.212+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:27.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:27 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:27.212+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:27.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:27 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:27.210+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:27.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:27 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:27.210+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:27.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:27 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:27.210+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:27.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:27.130+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:27.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:27.130+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:27.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:27.130+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.538066+0000 front 2024-05-03T17:39:57.538215+0000 (oldest deadline 2024-05-03T17:40:19.838038+0000) 2024-05-03T17:40:27.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:27.150+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:27.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:27.150+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:27.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:27.150+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:27.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:27.150+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:28.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:27 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:27.842+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:28.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:27 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:27.842+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:28.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:27 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:27.842+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:28.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:27 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:27.842+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:28.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:28 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:28.191+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:28.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:28 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:28.191+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:28.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:28 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:28.191+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:28.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:28 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:28.200+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:28.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:28 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:28.200+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:28.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:28 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:28.200+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:28.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:28 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:28.200+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:28.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:28 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:28.152+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:28.555 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:28 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:28.152+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:28.555 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:28 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:28.152+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.538066+0000 front 2024-05-03T17:39:57.538215+0000 (oldest deadline 2024-05-03T17:40:19.838038+0000) 2024-05-03T17:40:28.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:28 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:28.110+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:28.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:28 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:28.110+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:28.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:28 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:28.110+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:28.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:28 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:28.110+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:28.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:28 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:28.110+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:29.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:28 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:28.829+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:29.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:28 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:28.829+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:29.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:28 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:28.829+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:29.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:28 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:28.829+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:29.510 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:29 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:29.175+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:29.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:29 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:29.175+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:29.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:29 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:29.175+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:29.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:29 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:29.159+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:29.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:29 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:29.159+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:29.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:29 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:29.159+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:29.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:29 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:29.159+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:29.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:29 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:29.156+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:29.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:29 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:29.156+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:29.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:29 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:29.156+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.538066+0000 front 2024-05-03T17:39:57.538215+0000 (oldest deadline 2024-05-03T17:40:19.838038+0000) 2024-05-03T17:40:29.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:29 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:29.156+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:06.239558+0000 front 2024-05-03T17:40:06.239609+0000 (oldest deadline 2024-05-03T17:40:29.139656+0000) 2024-05-03T17:40:29.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:29 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:29.136+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:29.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:29 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:29.136+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:29.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:29 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:29.136+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:29.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:29 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:29.136+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:29.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:29 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:29.136+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:30.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:29 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:29.867+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:30.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:29 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:29.867+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:30.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:29 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:29.867+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:30.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:29 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:29.867+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:30.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:30 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:30.170+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:30.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:30 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:30.170+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:30.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:30 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:30.170+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:30.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:30 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:30.130+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:30.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:30 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:30.130+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:30.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:30 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:30.130+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:30.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:30 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:30.130+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:30.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:30 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:30.149+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:30.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:30 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:30.149+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:30.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:30 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:30.149+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.538066+0000 front 2024-05-03T17:39:57.538215+0000 (oldest deadline 2024-05-03T17:40:19.838038+0000) 2024-05-03T17:40:30.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:30 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:30.149+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:06.239558+0000 front 2024-05-03T17:40:06.239609+0000 (oldest deadline 2024-05-03T17:40:29.139656+0000) 2024-05-03T17:40:30.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:30 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:30.179+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:30.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:30 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:30.179+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:30.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:30 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:30.179+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:30.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:30 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:30.179+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:30.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:30 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:30.179+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:31.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:30 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:30.877+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:31.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:30 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:30.877+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:31.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:30 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:30.877+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:31.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:30 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:30.877+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:31.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:30 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:30.877+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:31.510 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:31 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:31.147+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:31.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:31 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:31.147+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:31.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:31 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:31.147+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:31.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:31 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:31.084+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:31.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:31 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:31.084+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:31.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:31 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:31.084+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:31.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:31 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:31.084+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:31.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:31 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:31.183+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:40.734412+0000 front 2024-05-03T17:39:40.734348+0000 (oldest deadline 2024-05-03T17:40:06.034480+0000) 2024-05-03T17:40:31.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:31 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:31.183+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:50.536923+0000 front 2024-05-03T17:39:50.536785+0000 (oldest deadline 2024-05-03T17:40:11.636866+0000) 2024-05-03T17:40:31.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:31 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:31.183+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.538066+0000 front 2024-05-03T17:39:57.538215+0000 (oldest deadline 2024-05-03T17:40:19.838038+0000) 2024-05-03T17:40:31.555 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:31 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7[51748]: 2024-05-03T17:40:31.183+0000 7fd8b1d21700 -1 osd.7 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:06.239558+0000 front 2024-05-03T17:40:06.239609+0000 (oldest deadline 2024-05-03T17:40:29.139656+0000) 2024-05-03T17:40:31.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:31 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:31.143+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:31.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:31 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:31.143+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:31.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:31 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:31.143+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:31.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:31 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:31.143+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:31.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:31 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:31.143+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:31.881 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:31 smithi070 podman[69124]: 2024-05-03 17:40:31.582752568 +0000 UTC m=+5.514060566 container died 3ed45e5cc559c8351229c2d50bb3ad3a9ed1e5c922c15b87f961629edc2f96fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-03T17:40:32.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:31 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:31.846+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:32.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:31 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:31.846+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:32.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:31 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:31.846+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:32.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:31 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:31.846+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:32.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:31 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:31.846+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:32.239 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:31 smithi070 podman[69124]: 2024-05-03 17:40:31.967784625 +0000 UTC m=+5.899092630 container cleanup 3ed45e5cc559c8351229c2d50bb3ad3a9ed1e5c922c15b87f961629edc2f96fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-03T17:40:32.240 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:31 smithi070 bash[69124]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7 2024-05-03T17:40:32.240 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:32 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:32.113+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:32.240 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:32 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:32.113+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:32.240 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:32 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:32.113+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:32.240 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:32 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:32.113+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:32.240 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:32 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:32.113+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:32.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:32 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:32.180+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:32.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:32 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:32.180+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:32.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:32 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:32.180+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:32.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:32 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:32.180+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:06.265704+0000 front 2024-05-03T17:40:06.265763+0000 (oldest deadline 2024-05-03T17:40:31.565375+0000) 2024-05-03T17:40:32.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:32 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:32.089+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:32.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:32 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:32.089+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:32.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:32 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:32.089+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:32.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:32 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:32.089+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:32.554 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:32 smithi070 podman[69141]: 2024-05-03 17:40:32.243945676 +0000 UTC m=+0.656629900 container remove 3ed45e5cc559c8351229c2d50bb3ad3a9ed1e5c922c15b87f961629edc2f96fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-03T17:40:32.860 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:32 smithi070 podman[69233]: 2024-05-03 17:40:32.482620905 +0000 UTC m=+0.017702357 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:40:32.861 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:32 smithi070 podman[69233]: 2024-05-03 17:40:32.594918052 +0000 UTC m=+0.129999486 container create d5d82aa9631fe5ce9de64c49bb8893477f2d9dfc719780765ba5517fa58ae49e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, ceph=True) 2024-05-03T17:40:32.861 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:32 smithi070 podman[69233]: 2024-05-03 17:40:32.864098944 +0000 UTC m=+0.399180377 container init d5d82aa9631fe5ce9de64c49bb8893477f2d9dfc719780765ba5517fa58ae49e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7-deactivate, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:40:33.152 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:32 smithi070 podman[69233]: 2024-05-03 17:40:32.867948492 +0000 UTC m=+0.403029928 container start d5d82aa9631fe5ce9de64c49bb8893477f2d9dfc719780765ba5517fa58ae49e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-03T17:40:33.152 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:32 smithi070 podman[69233]: 2024-05-03 17:40:32.930796738 +0000 UTC m=+0.465878189 container attach d5d82aa9631fe5ce9de64c49bb8893477f2d9dfc719780765ba5517fa58ae49e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-05-03T17:40:33.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:32 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:32.829+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:33.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:32 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:32.829+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:33.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:32 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:32.829+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:33.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:32 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:32.829+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:33.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:32 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:32.829+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:33.447 INFO:journalctl@ceph.osd.7.smithi070.stdout:May 03 17:40:33 smithi070 podman[69233]: 2024-05-03 17:40:33.162329555 +0000 UTC m=+0.697410986 container died d5d82aa9631fe5ce9de64c49bb8893477f2d9dfc719780765ba5517fa58ae49e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5) 2024-05-03T17:40:33.448 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:33 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:33.155+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:33.448 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:33 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:33.155+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:33.448 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:33 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:33.155+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:33.448 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:33 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:33.155+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:33.449 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:33 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:33.155+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:33.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:33 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:33.189+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:33.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:33 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:33.189+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:33.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:33 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:33.189+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:33.512 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:33 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:33.189+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:06.265704+0000 front 2024-05-03T17:40:06.265763+0000 (oldest deadline 2024-05-03T17:40:31.565375+0000) 2024-05-03T17:40:33.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:33 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:33.104+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:33.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:33 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:33.104+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:33.513 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:33 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:33.104+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:33.513 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:33 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:33.104+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:33.761 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.7.service' 2024-05-03T17:40:33.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:40:33.799 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-03T17:40:33.800 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-03T17:40:33.800 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.8 2024-05-03T17:40:34.099 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:33 smithi082 systemd[1]: Stopping Ceph osd.8 for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:40:34.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:33 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:33.821+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:34.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:33 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:33.821+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:34.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:33 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:33.821+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:34.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:33 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:33.821+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:34.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:33 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:33.821+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:34.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:34.122+0000 7fcc4127a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:40:34.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:34.122+0000 7fcc4127a700 -1 osd.8 84 *** Got signal Terminated *** 2024-05-03T17:40:34.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:34.123+0000 7fcc4127a700 -1 osd.8 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T17:40:34.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:34.170+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:34.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:34.170+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:34.512 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:34.170+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:34.512 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:34.170+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:06.265704+0000 front 2024-05-03T17:40:06.265763+0000 (oldest deadline 2024-05-03T17:40:31.565375+0000) 2024-05-03T17:40:34.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:34.102+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:34.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:34.102+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:34.513 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:34.102+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:34.513 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:34 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:34.102+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:34.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:34 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:34.114+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:34.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:34 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:34.115+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:34.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:34 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:34.115+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:34.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:34 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:34.115+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:34.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:34 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:34.115+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:35.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:34 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:34.836+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:35.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:34 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:34.836+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:35.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:34 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:34.836+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:35.201 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:34 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:34.836+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:35.201 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:34 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:34.836+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:35.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:35 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:35.212+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:35.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:35 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:35.212+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:35.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:35 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:35.212+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:35.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:35 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:35.212+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:06.265704+0000 front 2024-05-03T17:40:06.265763+0000 (oldest deadline 2024-05-03T17:40:31.565375+0000) 2024-05-03T17:40:35.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:35 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:35.126+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:35.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:35 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:35.126+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:35.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:35 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:35.126+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:35.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:35 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:35.126+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:35.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:35 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:35.073+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:35.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:35 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:35.073+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:35.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:35 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:35.073+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:35.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:35 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:35.073+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:35.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:35 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:35.073+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:36.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:35 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:35.880+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:36.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:35 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:35.880+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:36.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:35 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:35.880+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:36.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:35 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:35.880+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:36.201 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:35 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:35.880+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:36.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:36 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:36.037+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:36.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:36 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:36.037+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:36.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:36 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:36.037+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:36.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:36 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:36.037+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:36.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:36 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:36.037+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:36.412 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:36 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:36.207+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:36.412 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:36 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:36.207+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:36.412 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:36 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:36.207+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:36.412 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:36 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:36.207+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:06.265704+0000 front 2024-05-03T17:40:06.265763+0000 (oldest deadline 2024-05-03T17:40:31.565375+0000) 2024-05-03T17:40:36.413 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:36 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:36.087+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:36.413 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:36 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:36.087+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:36.413 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:36 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:36.087+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:36.413 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:36 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:36.087+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:36.844 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:36 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:36.846+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:36.844 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:36 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:36.846+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:36.844 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:36 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:36.846+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:36.845 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:36 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:36.846+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:36.845 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:36 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:36.846+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:36.845 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:36 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:36.846+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:11.799647+0000 front 2024-05-03T17:40:11.799592+0000 (oldest deadline 2024-05-03T17:40:35.899589+0000) 2024-05-03T17:40:37.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:37 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:37.009+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:37.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:37 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:37.009+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:37.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:37 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:37.009+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:37.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:37 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:37.009+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:37.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:37 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:37.009+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:37.510 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:37 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:37.225+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:37.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:37 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:37.225+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:37.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:37 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:37.225+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:37.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:37 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:37.225+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:06.265704+0000 front 2024-05-03T17:40:06.265763+0000 (oldest deadline 2024-05-03T17:40:31.565375+0000) 2024-05-03T17:40:37.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:37 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:37.109+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:37.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:37 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:37.109+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:37.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:37 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:37.109+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:37.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:37 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:37.109+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:38.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:37 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:37.833+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:38.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:37 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:37.833+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:38.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:37 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:37.833+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:38.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:37 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:37.833+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:38.201 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:37 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:37.833+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:38.201 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:37 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:37.833+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:11.799647+0000 front 2024-05-03T17:40:11.799592+0000 (oldest deadline 2024-05-03T17:40:35.899589+0000) 2024-05-03T17:40:38.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:37 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:37.969+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:38.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:37 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:37.969+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:38.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:37 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:37.969+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:38.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:37 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:37.969+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:38.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:37 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:37.969+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:38.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:37 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:37.969+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:38.510 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:38 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:38.225+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:42.560291+0000 front 2024-05-03T17:39:42.560049+0000 (oldest deadline 2024-05-03T17:40:07.260048+0000) 2024-05-03T17:40:38.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:38 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:38.225+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.861891+0000 front 2024-05-03T17:39:48.861874+0000 (oldest deadline 2024-05-03T17:40:12.962016+0000) 2024-05-03T17:40:38.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:38 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:38.225+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:54.563595+0000 front 2024-05-03T17:39:54.563605+0000 (oldest deadline 2024-05-03T17:40:20.463583+0000) 2024-05-03T17:40:38.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:38 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:38.225+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:06.265704+0000 front 2024-05-03T17:40:06.265763+0000 (oldest deadline 2024-05-03T17:40:31.565375+0000) 2024-05-03T17:40:38.511 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:38 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8[52649]: 2024-05-03T17:40:38.225+0000 7fcc3d86e700 -1 osd.8 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:11.565705+0000 front 2024-05-03T17:40:11.565683+0000 (oldest deadline 2024-05-03T17:40:37.465900+0000) 2024-05-03T17:40:38.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:38 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:38.077+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:38.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:38 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:38.077+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:38.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:38 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:38.077+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:38.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:38 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:38.077+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:38.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:38 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:38.077+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:39.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:38 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:38.837+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:39.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:38 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:38.838+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:39.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:38 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:38.838+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:39.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:38 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:38.838+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:39.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:38 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:38.838+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:39.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:38 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:38.838+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:11.799647+0000 front 2024-05-03T17:40:11.799592+0000 (oldest deadline 2024-05-03T17:40:35.899589+0000) 2024-05-03T17:40:39.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:38 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:38.927+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:39.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:38 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:38.927+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:39.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:38 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:38.927+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:39.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:38 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:38.927+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:39.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:38 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:38.927+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:39.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:38 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:38.927+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:39.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:39 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:39.042+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:39.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:39 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:39.042+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:39.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:39 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:39.042+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:39.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:39 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:39.042+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:39.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:39 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:39.042+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:39.844 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:39 smithi082 podman[70313]: 2024-05-03 17:40:39.548625425 +0000 UTC m=+5.561512710 container died c3fcbbe3a3178d1f10f26a6f57f1e34044021f2db287599c5d895dded214b73a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430) 2024-05-03T17:40:40.186 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:39 smithi082 podman[70313]: 2024-05-03 17:40:39.939638085 +0000 UTC m=+5.952525359 container cleanup c3fcbbe3a3178d1f10f26a6f57f1e34044021f2db287599c5d895dded214b73a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-03T17:40:40.187 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:39 smithi082 bash[70313]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8 2024-05-03T17:40:40.187 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:40 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:40.059+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:40.187 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:40 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:40.059+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:40.187 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:40 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:40.059+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:40.187 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:40 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:40.059+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:40.188 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:40 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:40.059+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:40.201 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:39 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:39.865+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:40.202 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:39 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:39.865+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:40.202 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:39 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:39.865+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:40.202 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:39 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:39.865+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:40.202 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:39 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:39.865+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:40.202 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:39 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:39.865+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:11.799647+0000 front 2024-05-03T17:40:11.799592+0000 (oldest deadline 2024-05-03T17:40:35.899589+0000) 2024-05-03T17:40:40.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:39 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:39.973+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:40.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:39 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:39.973+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:40.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:39 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:39.973+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:40.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:39 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:39.973+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:40.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:39 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:39.973+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:40.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:39 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:39.973+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:40.485 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:40 smithi082 podman[70342]: 2024-05-03 17:40:40.190770639 +0000 UTC m=+0.636141914 container remove c3fcbbe3a3178d1f10f26a6f57f1e34044021f2db287599c5d895dded214b73a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:40:40.761 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:40 smithi082 podman[70434]: 2024-05-03 17:40:40.392167443 +0000 UTC m=+0.017879028 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:40:40.761 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:40 smithi082 podman[70434]: 2024-05-03 17:40:40.49162983 +0000 UTC m=+0.117341440 container create 0432c8dab690dc6a5018f728e1cf1da33124efb53296318667ac58ed2898d61d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-03T17:40:40.761 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:40 smithi082 podman[70434]: 2024-05-03 17:40:40.718976046 +0000 UTC m=+0.344687616 container init 0432c8dab690dc6a5018f728e1cf1da33124efb53296318667ac58ed2898d61d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-03T17:40:40.761 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:40 smithi082 podman[70434]: 2024-05-03 17:40:40.721914606 +0000 UTC m=+0.347626174 container start 0432c8dab690dc6a5018f728e1cf1da33124efb53296318667ac58ed2898d61d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-03T17:40:41.030 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:40 smithi082 podman[70434]: 2024-05-03 17:40:40.827794269 +0000 UTC m=+0.453505846 container attach 0432c8dab690dc6a5018f728e1cf1da33124efb53296318667ac58ed2898d61d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8-deactivate, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:40:41.031 INFO:journalctl@ceph.osd.8.smithi082.stdout:May 03 17:40:41 smithi082 podman[70434]: 2024-05-03 17:40:41.012535198 +0000 UTC m=+0.638246759 container died 0432c8dab690dc6a5018f728e1cf1da33124efb53296318667ac58ed2898d61d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-8-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-03T17:40:41.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:40 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:40.855+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:41.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:40 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:40.855+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:41.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:40 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:40.855+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:41.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:40 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:40.855+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:41.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:40 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:40.855+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:41.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:40 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:40.855+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:11.799647+0000 front 2024-05-03T17:40:11.799592+0000 (oldest deadline 2024-05-03T17:40:35.899589+0000) 2024-05-03T17:40:41.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:41 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:41.021+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:41.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:41 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:41.021+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:41.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:41 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:41.021+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:41.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:41 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:41.021+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:41.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:41 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:41.021+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:41.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:41 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:41.021+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:41.386 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:41 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:41.109+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:41.387 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:41 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:41.109+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:41.387 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:41 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:41.109+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:41.387 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:41 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:41.109+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:41.387 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:41 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:41.109+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:41.723 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.8.service' 2024-05-03T17:40:41.764 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:40:41.765 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-03T17:40:41.765 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-03T17:40:41.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.9 2024-05-03T17:40:42.101 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:41 smithi067 systemd[1]: Stopping Ceph osd.9 for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:40:42.102 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:41 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:41.862+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:42.102 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:41 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:41.862+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:42.102 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:41 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:41.862+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:42.102 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:41 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:41.862+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:42.102 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:41 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:41.862+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:42.102 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:41 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:41.862+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:11.799647+0000 front 2024-05-03T17:40:11.799592+0000 (oldest deadline 2024-05-03T17:40:35.899589+0000) 2024-05-03T17:40:42.102 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:42 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:42.105+0000 7f3e32874700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:40:42.103 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:42 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:42.105+0000 7f3e32874700 -1 osd.9 84 *** Got signal Terminated *** 2024-05-03T17:40:42.103 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:42 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:42.105+0000 7f3e32874700 -1 osd.9 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T17:40:42.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:42 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:42.057+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:42.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:42 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:42.057+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:42.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:42 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:42.057+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:42.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:42 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:42.057+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:42.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:42 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:42.057+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:42.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:42 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:42.057+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:42.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:42 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:42.083+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:42.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:42 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:42.083+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:42.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:42 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:42.083+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:42.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:42 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:42.083+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:42.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:42 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:42.083+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:43.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:42 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:42.812+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:43.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:42 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:42.812+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:43.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:42 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:42.812+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:43.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:42 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:42.812+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:43.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:42 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:42.812+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:43.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:42 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:42.812+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:11.799647+0000 front 2024-05-03T17:40:11.799592+0000 (oldest deadline 2024-05-03T17:40:35.899589+0000) 2024-05-03T17:40:43.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:43 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:43.133+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:43.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:43 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:43.133+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:43.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:43 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:43.133+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:43.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:43 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:43.133+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:43.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:43 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:43.133+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:43.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:43 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:43.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:43.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:43 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:43.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:43.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:43 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:43.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:43.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:43 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:43.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:43.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:43 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:43.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:43.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:43 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:43.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:44.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:43.843+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:44.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:43.843+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:44.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:43.843+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:44.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:43.843+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:44.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:43.843+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:44.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:43 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:43.843+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:11.799647+0000 front 2024-05-03T17:40:11.799592+0000 (oldest deadline 2024-05-03T17:40:35.899589+0000) 2024-05-03T17:40:44.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:44 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:44.157+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:44.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:44 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:44.157+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:44.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:44 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:44.157+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:44.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:44 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:44.157+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:44.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:44 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:44.157+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:44.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:44 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:44.083+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:44.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:44 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:44.083+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:44.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:44 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:44.083+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:44.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:44 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:44.083+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:44.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:44 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:44.083+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:44.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:44 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:44.083+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:45.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:44 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:44.810+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:45.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:44 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:44.810+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:45.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:44 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:44.810+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:45.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:44 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:44.810+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:45.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:44 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:44.810+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:45.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:44 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:44.810+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:11.799647+0000 front 2024-05-03T17:40:11.799592+0000 (oldest deadline 2024-05-03T17:40:35.899589+0000) 2024-05-03T17:40:45.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:44 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:44.810+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:18.800486+0000 front 2024-05-03T17:40:18.800530+0000 (oldest deadline 2024-05-03T17:40:44.100570+0000) 2024-05-03T17:40:45.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:45 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:45.053+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:45.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:45 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:45.053+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:45.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:45 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:45.053+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:45.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:45 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:45.053+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:45.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:45 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:45.053+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:45.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:45 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:45.053+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:45.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:45 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:45.053+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:20.985383+0000 front 2024-05-03T17:40:20.985488+0000 (oldest deadline 2024-05-03T17:40:44.485311+0000) 2024-05-03T17:40:45.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:45 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:45.144+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:45.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:45 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:45.144+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:45.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:45 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:45.144+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:45.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:45 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:45.144+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:45.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:45 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:45.144+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:45.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:45 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:45.144+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:46.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:45 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:45.780+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:46.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:45 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:45.780+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:46.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:45 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:45.780+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:46.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:45 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:45.780+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:46.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:45 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:45.780+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:46.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:45 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:45.780+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:11.799647+0000 front 2024-05-03T17:40:11.799592+0000 (oldest deadline 2024-05-03T17:40:35.899589+0000) 2024-05-03T17:40:46.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:45 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:45.780+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:18.800486+0000 front 2024-05-03T17:40:18.800530+0000 (oldest deadline 2024-05-03T17:40:44.100570+0000) 2024-05-03T17:40:46.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:46 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:46.114+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:46.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:46 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:46.114+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:46.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:46 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:46.114+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:46.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:46 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:46.114+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:46.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:46 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:46.114+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:46.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:46 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:46.114+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:46.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:46 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:46.098+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:46.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:46 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:46.098+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:46.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:46 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:46.098+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:46.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:46 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:46.098+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:46.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:46 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:46.098+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:46.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:46 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:46.098+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:46.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:46 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:46.098+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:20.985383+0000 front 2024-05-03T17:40:20.985488+0000 (oldest deadline 2024-05-03T17:40:44.485311+0000) 2024-05-03T17:40:47.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:46 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:46.784+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:32.594188+0000 front 2024-05-03T17:39:32.594233+0000 (oldest deadline 2024-05-03T17:39:57.894356+0000) 2024-05-03T17:40:47.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:46 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:46.784+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:37.894748+0000 front 2024-05-03T17:39:37.894807+0000 (oldest deadline 2024-05-03T17:40:03.794891+0000) 2024-05-03T17:40:47.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:46 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:46.784+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:48.996656+0000 front 2024-05-03T17:39:48.996787+0000 (oldest deadline 2024-05-03T17:40:13.096789+0000) 2024-05-03T17:40:47.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:46 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:46.784+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.197440+0000 front 2024-05-03T17:39:57.197480+0000 (oldest deadline 2024-05-03T17:40:23.097669+0000) 2024-05-03T17:40:47.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:46 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:46.784+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:05.998816+0000 front 2024-05-03T17:40:05.999017+0000 (oldest deadline 2024-05-03T17:40:30.698618+0000) 2024-05-03T17:40:47.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:46 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:46.784+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:11.799647+0000 front 2024-05-03T17:40:11.799592+0000 (oldest deadline 2024-05-03T17:40:35.899589+0000) 2024-05-03T17:40:47.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:46 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9[65397]: 2024-05-03T17:40:46.784+0000 7f3e2ae60700 -1 osd.9 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:18.800486+0000 front 2024-05-03T17:40:18.800530+0000 (oldest deadline 2024-05-03T17:40:44.100570+0000) 2024-05-03T17:40:47.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:47 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:47.088+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:47.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:47 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:47.089+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:47.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:47 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:47.089+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:47.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:47 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:47.089+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:47.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:47 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:47.089+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:47.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:47 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:47.089+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:47.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:47 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:47.089+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:20.985383+0000 front 2024-05-03T17:40:20.985488+0000 (oldest deadline 2024-05-03T17:40:44.485311+0000) 2024-05-03T17:40:47.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:47 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:47.075+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:47.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:47 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:47.075+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:47.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:47 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:47.075+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:47.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:47 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:47.075+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:47.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:47 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:47.075+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:47.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:47 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:47.075+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:47.699 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:47 smithi067 podman[91505]: 2024-05-03 17:40:47.438882375 +0000 UTC m=+5.471386172 container died 63149a62cb1b6b2f6cc4aaca87d53eb57386af142d0affa3b1b949219005535e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-05-03T17:40:48.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:47 smithi067 podman[91505]: 2024-05-03 17:40:47.779055719 +0000 UTC m=+5.811559511 container cleanup 63149a62cb1b6b2f6cc4aaca87d53eb57386af142d0affa3b1b949219005535e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=) 2024-05-03T17:40:48.199 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:47 smithi067 bash[91505]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9 2024-05-03T17:40:48.200 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:47 smithi067 podman[91520]: 2024-05-03 17:40:47.94675333 +0000 UTC m=+0.504971874 container remove 63149a62cb1b6b2f6cc4aaca87d53eb57386af142d0affa3b1b949219005535e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:40:48.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:48 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:48.056+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:48.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:48 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:48.056+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:48.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:48 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:48.056+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:48.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:48 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:48.056+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:48.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:48 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:48.056+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:48.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:48 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:48.056+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:48.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:48 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:48.056+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:20.985383+0000 front 2024-05-03T17:40:20.985488+0000 (oldest deadline 2024-05-03T17:40:44.485311+0000) 2024-05-03T17:40:48.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:48 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:48.061+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:48.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:48 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:48.061+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:48.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:48 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:48.061+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:48.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:48 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:48.061+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:48.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:48 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:48.061+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:48.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:48 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:48.061+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:48.699 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:48 smithi067 podman[91616]: 2024-05-03 17:40:48.306596834 +0000 UTC m=+0.018651414 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:40:48.699 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:48 smithi067 podman[91616]: 2024-05-03 17:40:48.422666975 +0000 UTC m=+0.134721535 container create c4d7e6c165f370ec7c35f3300c5097f1cf6baf780a1a390392d9e56d94ff427c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, io.buildah.version=1.33.5) 2024-05-03T17:40:48.700 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:48 smithi067 podman[91616]: 2024-05-03 17:40:48.641663343 +0000 UTC m=+0.353717905 container init c4d7e6c165f370ec7c35f3300c5097f1cf6baf780a1a390392d9e56d94ff427c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-03T17:40:48.700 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:48 smithi067 podman[91616]: 2024-05-03 17:40:48.644786422 +0000 UTC m=+0.356840981 container start c4d7e6c165f370ec7c35f3300c5097f1cf6baf780a1a390392d9e56d94ff427c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-05-03T17:40:49.202 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:48 smithi067 podman[91616]: 2024-05-03 17:40:48.733098016 +0000 UTC m=+0.445152586 container attach c4d7e6c165f370ec7c35f3300c5097f1cf6baf780a1a390392d9e56d94ff427c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-05-03T17:40:49.202 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:48 smithi067 podman[91616]: 2024-05-03 17:40:48.953564888 +0000 UTC m=+0.665619452 container died c4d7e6c165f370ec7c35f3300c5097f1cf6baf780a1a390392d9e56d94ff427c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-03T17:40:49.495 INFO:journalctl@ceph.osd.9.smithi067.stdout:May 03 17:40:49 smithi067 podman[91641]: 2024-05-03 17:40:49.486567217 +0000 UTC m=+0.526603591 container remove c4d7e6c165f370ec7c35f3300c5097f1cf6baf780a1a390392d9e56d94ff427c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-03T17:40:49.501 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.9.service' 2024-05-03T17:40:49.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:49 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:49.079+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:49.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:49 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:49.079+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:49.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:49 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:49.079+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:49.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:49 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:49.079+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:49.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:49 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:49.079+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:49.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:49 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:49.079+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:49.540 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:40:49.540 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-03T17:40:49.540 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-03T17:40:49.541 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.10 2024-05-03T17:40:49.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:49 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:49.104+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:49.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:49 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:49.104+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:49.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:49 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:49.104+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:49.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:49 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:49.104+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:49.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:49 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:49.104+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:49.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:49 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:49.104+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:49.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:49 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:49.104+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:20.985383+0000 front 2024-05-03T17:40:20.985488+0000 (oldest deadline 2024-05-03T17:40:44.485311+0000) 2024-05-03T17:40:49.832 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:49 smithi070 systemd[1]: Stopping Ceph osd.10 for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:40:50.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:49 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:49.834+0000 7f98a5fef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:40:50.304 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:49 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:49.834+0000 7f98a5fef700 -1 osd.10 84 *** Got signal Terminated *** 2024-05-03T17:40:50.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:49 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:49.834+0000 7f98a5fef700 -1 osd.10 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T17:40:50.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:50 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:50.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:50.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:50 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:50.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:50.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:50 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:50.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:50.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:50 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:50.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:50.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:50 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:50.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:50.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:50 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:50.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:50.305 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:50 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:50.079+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:20.985383+0000 front 2024-05-03T17:40:20.985488+0000 (oldest deadline 2024-05-03T17:40:44.485311+0000) 2024-05-03T17:40:50.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:50 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:50.087+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:50.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:50 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:50.087+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:50.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:50 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:50.087+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:50.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:50 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:50.087+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:50.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:50 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:50.087+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:50.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:50 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:50.087+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:51.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:51 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:51.107+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:51.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:51 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:51.107+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:51.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:51 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:51.107+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:51.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:51 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:51.107+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:51.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:51 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:51.107+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:51.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:51 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:51.107+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:51.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:51 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:51.116+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:51.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:51 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:51.116+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:51.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:51 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:51.116+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:51.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:51 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:51.116+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:51.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:51 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:51.116+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:51.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:51 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:51.116+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:51.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:51 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:51.116+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:20.985383+0000 front 2024-05-03T17:40:20.985488+0000 (oldest deadline 2024-05-03T17:40:44.485311+0000) 2024-05-03T17:40:52.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:52 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:52.112+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:52.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:52 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:52.112+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:52.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:52 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:52.112+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:52.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:52 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:52.112+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:52.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:52 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:52.112+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:52.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:52 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:52.112+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:52.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:52 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:52.132+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:52.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:52 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:52.132+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:52.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:52 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:52.132+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:52.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:52 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:52.132+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:52.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:52 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:52.132+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:52.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:52 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:52.132+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:52.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:52 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:52.132+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:20.985383+0000 front 2024-05-03T17:40:20.985488+0000 (oldest deadline 2024-05-03T17:40:44.485311+0000) 2024-05-03T17:40:53.513 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:53 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:53.140+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:53.514 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:53 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:53.140+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:53.514 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:53 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:53.140+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:53.514 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:53 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:53.140+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:53.514 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:53 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:53.140+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:53.514 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:53 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:53.140+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:53.514 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:53 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:53.140+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-05-03T17:40:28.971327+0000 front 2024-05-03T17:40:28.971355+0000 (oldest deadline 2024-05-03T17:40:53.071314+0000) 2024-05-03T17:40:53.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:53 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:53.161+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:53.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:53 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:53.161+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:53.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:53 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:53.161+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:53.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:53 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:53.161+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:53.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:53 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:53.161+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:53.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:53 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:53.161+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:53.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:53 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:53.161+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:20.985383+0000 front 2024-05-03T17:40:20.985488+0000 (oldest deadline 2024-05-03T17:40:44.485311+0000) 2024-05-03T17:40:54.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:54 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:54.179+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:54.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:54 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:54.179+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:54.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:54 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:54.179+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:54.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:54 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:54.179+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:54.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:54 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:54.179+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:54.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:54 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:54.179+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:54.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:54 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:54.179+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-05-03T17:40:28.971327+0000 front 2024-05-03T17:40:28.971355+0000 (oldest deadline 2024-05-03T17:40:53.071314+0000) 2024-05-03T17:40:54.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:54 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:54.163+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:34.377532+0000 front 2024-05-03T17:39:34.377661+0000 (oldest deadline 2024-05-03T17:39:59.077504+0000) 2024-05-03T17:40:54.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:54 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:54.163+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:39.578521+0000 front 2024-05-03T17:39:39.578489+0000 (oldest deadline 2024-05-03T17:40:05.478678+0000) 2024-05-03T17:40:54.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:54 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:54.163+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.479303+0000 front 2024-05-03T17:39:45.479331+0000 (oldest deadline 2024-05-03T17:40:13.679850+0000) 2024-05-03T17:40:54.554 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:54 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:54.163+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6814 osd.3 since back 2024-05-03T17:39:57.181192+0000 front 2024-05-03T17:39:57.181278+0000 (oldest deadline 2024-05-03T17:40:20.681253+0000) 2024-05-03T17:40:54.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:54 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:54.164+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:04.782121+0000 front 2024-05-03T17:40:04.782138+0000 (oldest deadline 2024-05-03T17:40:27.682174+0000) 2024-05-03T17:40:54.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:54 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:54.164+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:10.483654+0000 front 2024-05-03T17:40:10.483672+0000 (oldest deadline 2024-05-03T17:40:37.484198+0000) 2024-05-03T17:40:54.555 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:54 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10[58480]: 2024-05-03T17:40:54.164+0000 7f989e5db700 -1 osd.10 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:20.985383+0000 front 2024-05-03T17:40:20.985488+0000 (oldest deadline 2024-05-03T17:40:44.485311+0000) 2024-05-03T17:40:55.507 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:55 smithi070 podman[69351]: 2024-05-03 17:40:55.248102306 +0000 UTC m=+5.510160515 container died 0cf0da3eed518cef329dbcdf2fdf98939fd300a9b1f0a7e49919587b42ab1f11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:40:55.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:55 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:55.210+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:55.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:55 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:55.210+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:55.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:55 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:55.210+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:55.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:55 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:55.210+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:55.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:55 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:55.210+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:55.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:55 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:55.210+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:55.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:55 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:55.210+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-05-03T17:40:28.971327+0000 front 2024-05-03T17:40:28.971355+0000 (oldest deadline 2024-05-03T17:40:53.071314+0000) 2024-05-03T17:40:55.804 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:55 smithi070 podman[69351]: 2024-05-03 17:40:55.585023534 +0000 UTC m=+5.847081741 container cleanup 0cf0da3eed518cef329dbcdf2fdf98939fd300a9b1f0a7e49919587b42ab1f11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:40:55.804 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:55 smithi070 bash[69351]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10 2024-05-03T17:40:55.804 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:55 smithi070 podman[69368]: 2024-05-03 17:40:55.744213331 +0000 UTC m=+0.494608615 container remove 0cf0da3eed518cef329dbcdf2fdf98939fd300a9b1f0a7e49919587b42ab1f11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-03T17:40:56.426 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:56 smithi070 podman[69456]: 2024-05-03 17:40:56.161722649 +0000 UTC m=+0.114764246 container create 606a25d49c42e5312af35abe64962a277f7ea0603ad58168ba521e491ecc6204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:40:56.427 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:56 smithi070 podman[69456]: 2024-05-03 17:40:56.065316943 +0000 UTC m=+0.018358546 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:40:56.427 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:56 smithi070 podman[69456]: 2024-05-03 17:40:56.356451435 +0000 UTC m=+0.309493034 container init 606a25d49c42e5312af35abe64962a277f7ea0603ad58168ba521e491ecc6204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True) 2024-05-03T17:40:56.427 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:56 smithi070 podman[69456]: 2024-05-03 17:40:56.359893164 +0000 UTC m=+0.312934758 container start 606a25d49c42e5312af35abe64962a277f7ea0603ad58168ba521e491ecc6204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-03T17:40:56.427 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:56 smithi070 podman[69456]: 2024-05-03 17:40:56.430828074 +0000 UTC m=+0.383869672 container attach 606a25d49c42e5312af35abe64962a277f7ea0603ad58168ba521e491ecc6204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_CLEAN=True) 2024-05-03T17:40:56.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:56 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:56.239+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:56.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:56 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:56.239+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:56.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:56 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:56.239+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:56.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:56 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:56.239+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:56.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:56 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:56.239+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:56.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:56 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:56.239+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:56.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:56 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:56.239+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-05-03T17:40:28.971327+0000 front 2024-05-03T17:40:28.971355+0000 (oldest deadline 2024-05-03T17:40:53.071314+0000) 2024-05-03T17:40:56.805 INFO:journalctl@ceph.osd.10.smithi070.stdout:May 03 17:40:56 smithi070 podman[69456]: 2024-05-03 17:40:56.654250715 +0000 UTC m=+0.607292307 container died 606a25d49c42e5312af35abe64962a277f7ea0603ad58168ba521e491ecc6204 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-05-03T17:40:57.208 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.10.service' 2024-05-03T17:40:57.289 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:40:57.289 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-03T17:40:57.289 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-03T17:40:57.289 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.11 2024-05-03T17:40:57.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:57 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:57.200+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:57.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:57 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:57.200+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:57.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:57 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:57.200+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:57.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:57 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:57.200+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:57.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:57 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:57.200+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:57.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:57 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:57.200+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:57.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:57 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:57.200+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-05-03T17:40:28.971327+0000 front 2024-05-03T17:40:28.971355+0000 (oldest deadline 2024-05-03T17:40:53.071314+0000) 2024-05-03T17:40:57.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:57 smithi082 systemd[1]: Stopping Ceph osd.11 for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:40:58.010 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:57 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:57.557+0000 7f594e759700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:40:58.011 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:57 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:57.557+0000 7f594e759700 -1 osd.11 84 *** Got signal Terminated *** 2024-05-03T17:40:58.011 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:57 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:57.557+0000 7f594e759700 -1 osd.11 84 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-03T17:40:58.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:58 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:58.225+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:58.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:58 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:58.225+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:58.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:58 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:58.225+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:58.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:58 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:58.225+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:58.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:58 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:58.225+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:58.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:58 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:58.225+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:58.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:58 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:58.225+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-05-03T17:40:28.971327+0000 front 2024-05-03T17:40:28.971355+0000 (oldest deadline 2024-05-03T17:40:53.071314+0000) 2024-05-03T17:40:59.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:59 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:59.191+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:40:59.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:59 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:59.191+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:40:59.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:59 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:59.191+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:40:59.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:59 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:59.191+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:40:59.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:59 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:59.191+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:40:59.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:59 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:59.191+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:40:59.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:40:59 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:40:59.191+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-05-03T17:40:28.971327+0000 front 2024-05-03T17:40:28.971355+0000 (oldest deadline 2024-05-03T17:40:53.071314+0000) 2024-05-03T17:41:00.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:00 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:00.204+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:41:00.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:00 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:00.204+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:41:00.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:00 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:00.204+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:41:00.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:00 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:00.204+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:41:00.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:00 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:00.204+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:41:00.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:00 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:00.204+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:41:00.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:00 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:00.204+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-05-03T17:40:28.971327+0000 front 2024-05-03T17:40:28.971355+0000 (oldest deadline 2024-05-03T17:40:53.071314+0000) 2024-05-03T17:41:00.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:00 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:00.204+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6820 osd.8 since back 2024-05-03T17:40:37.172130+0000 front 2024-05-03T17:40:37.172164+0000 (oldest deadline 2024-05-03T17:40:59.972976+0000) 2024-05-03T17:41:01.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:01 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:01.186+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:41:01.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:01 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:01.186+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:41:01.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:01 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:01.186+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:41:01.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:01 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:01.186+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:41:01.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:01 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:01.186+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:41:01.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:01 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:01.186+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:41:01.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:01 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:01.186+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-05-03T17:40:28.971327+0000 front 2024-05-03T17:40:28.971355+0000 (oldest deadline 2024-05-03T17:40:53.071314+0000) 2024-05-03T17:41:01.512 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:01 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:01.186+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6820 osd.8 since back 2024-05-03T17:40:37.172130+0000 front 2024-05-03T17:40:37.172164+0000 (oldest deadline 2024-05-03T17:40:59.972976+0000) 2024-05-03T17:41:02.510 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:02 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:02.154+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6806 osd.0 since back 2024-05-03T17:39:33.763560+0000 front 2024-05-03T17:39:33.763427+0000 (oldest deadline 2024-05-03T17:39:56.063315+0000) 2024-05-03T17:41:02.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:02 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:02.154+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6804 osd.1 since back 2024-05-03T17:39:43.064834+0000 front 2024-05-03T17:39:43.064715+0000 (oldest deadline 2024-05-03T17:40:05.964949+0000) 2024-05-03T17:41:02.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:02 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:02.154+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6804 osd.2 since back 2024-05-03T17:39:45.965435+0000 front 2024-05-03T17:39:45.965500+0000 (oldest deadline 2024-05-03T17:40:11.865420+0000) 2024-05-03T17:41:02.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:02 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:02.154+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6812 osd.4 since back 2024-05-03T17:40:03.467547+0000 front 2024-05-03T17:40:03.467533+0000 (oldest deadline 2024-05-03T17:40:27.567432+0000) 2024-05-03T17:41:02.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:02 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:02.154+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6812 osd.5 since back 2024-05-03T17:40:14.568855+0000 front 2024-05-03T17:40:14.568782+0000 (oldest deadline 2024-05-03T17:40:38.068877+0000) 2024-05-03T17:41:02.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:02 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:02.154+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.67:6822 osd.6 since back 2024-05-03T17:40:22.070234+0000 front 2024-05-03T17:40:22.070350+0000 (oldest deadline 2024-05-03T17:40:44.370035+0000) 2024-05-03T17:41:02.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:02 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:02.154+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.70:6820 osd.7 since back 2024-05-03T17:40:28.971327+0000 front 2024-05-03T17:40:28.971355+0000 (oldest deadline 2024-05-03T17:40:53.071314+0000) 2024-05-03T17:41:02.511 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:02 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11[59268]: 2024-05-03T17:41:02.154+0000 7f594ad4d700 -1 osd.11 84 heartbeat_check: no reply from 172.21.15.82:6820 osd.8 since back 2024-05-03T17:40:37.172130+0000 front 2024-05-03T17:40:37.172164+0000 (oldest deadline 2024-05-03T17:40:59.972976+0000) 2024-05-03T17:41:03.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:02 smithi082 podman[70540]: 2024-05-03 17:41:02.990360083 +0000 UTC m=+5.513492046 container died 71dfc6a2bf40ee9dc4a6483f47a898741ce7369cf89fe457cc9b4a80ef7746e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:41:03.537 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:03 smithi082 podman[70540]: 2024-05-03 17:41:03.365514141 +0000 UTC m=+5.888646098 container cleanup 71dfc6a2bf40ee9dc4a6483f47a898741ce7369cf89fe457cc9b4a80ef7746e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-03T17:41:03.537 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:03 smithi082 bash[70540]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11 2024-05-03T17:41:03.974 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:03 smithi082 podman[70558]: 2024-05-03 17:41:03.541555612 +0000 UTC m=+0.543839032 container remove 71dfc6a2bf40ee9dc4a6483f47a898741ce7369cf89fe457cc9b4a80ef7746e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:41:04.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:03 smithi082 podman[70651]: 2024-05-03 17:41:03.881311007 +0000 UTC m=+0.017430843 image pull ee4d846202496ba6096593e8e8becc0700a5eb7ebd95536031187f5f337d4012 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd 2024-05-03T17:41:04.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:03 smithi082 podman[70651]: 2024-05-03 17:41:03.992300394 +0000 UTC m=+0.128420219 container create 55a6fbd659f1814e6f2fd85f15c72a2b7fbf762e64184a0046c10cbb8cfa6474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11-deactivate, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-05-03T17:41:04.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:04 smithi082 podman[70651]: 2024-05-03 17:41:04.202811943 +0000 UTC m=+0.338931769 container init 55a6fbd659f1814e6f2fd85f15c72a2b7fbf762e64184a0046c10cbb8cfa6474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-05-03T17:41:04.261 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:04 smithi082 podman[70651]: 2024-05-03 17:41:04.205815517 +0000 UTC m=+0.341935343 container start 55a6fbd659f1814e6f2fd85f15c72a2b7fbf762e64184a0046c10cbb8cfa6474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:41:04.761 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:04 smithi082 podman[70651]: 2024-05-03 17:41:04.297282815 +0000 UTC m=+0.433402641 container attach 55a6fbd659f1814e6f2fd85f15c72a2b7fbf762e64184a0046c10cbb8cfa6474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD) 2024-05-03T17:41:04.761 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:04 smithi082 podman[70651]: 2024-05-03 17:41:04.498261489 +0000 UTC m=+0.634381315 container died 55a6fbd659f1814e6f2fd85f15c72a2b7fbf762e64184a0046c10cbb8cfa6474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-03T17:41:05.029 INFO:journalctl@ceph.osd.11.smithi082.stdout:May 03 17:41:05 smithi082 podman[70676]: 2024-05-03 17:41:05.01450476 +0000 UTC m=+0.511097757 container remove 55a6fbd659f1814e6f2fd85f15c72a2b7fbf762e64184a0046c10cbb8cfa6474 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-osd-11-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-03T17:41:05.035 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@osd.11.service' 2024-05-03T17:41:05.071 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:05.072 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-03T17:41:05.072 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-03T17:41:05.072 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.a 2024-05-03T17:41:05.384 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 17:41:05 smithi067 systemd[1]: Stopping Ceph mds.a for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:41:05.699 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 17:41:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-a[72455]: 2024-05-03T17:41:05.387+0000 7f8373dc3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:41:05.699 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 17:41:05 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-a[72455]: 2024-05-03T17:41:05.388+0000 7f8373dc3700 -1 mds.a *** got signal Terminated *** 2024-05-03T17:41:08.818 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 17:41:08 smithi067 podman[91720]: 2024-05-03 17:41:08.541423134 +0000 UTC m=+3.262419132 container died 5f4389aa2a4a2af93dab527f555527991ef35e88a3ff58e05236e81d7a471d41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-a, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430) 2024-05-03T17:41:09.199 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 17:41:08 smithi067 podman[91720]: 2024-05-03 17:41:08.921051393 +0000 UTC m=+3.642047391 container cleanup 5f4389aa2a4a2af93dab527f555527991ef35e88a3ff58e05236e81d7a471d41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-a, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-05-03T17:41:09.199 INFO:journalctl@ceph.mds.a.smithi067.stdout:May 03 17:41:08 smithi067 bash[91720]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-a 2024-05-03T17:41:09.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.a.service' 2024-05-03T17:41:09.539 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:09.539 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-03T17:41:09.540 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-03T17:41:09.540 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.d 2024-05-03T17:41:09.946 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 17:41:09 smithi067 systemd[1]: Stopping Ceph mds.d for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:41:10.199 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 17:41:09 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-d[71183]: 2024-05-03T17:41:09.949+0000 7fd04617a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:41:10.199 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 17:41:09 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-d[71183]: 2024-05-03T17:41:09.949+0000 7fd04617a700 -1 mds.d *** got signal Terminated *** 2024-05-03T17:41:14.346 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 17:41:13 smithi067 podman[91854]: 2024-05-03 17:41:13.958901489 +0000 UTC m=+4.190044911 container died 3829824985cba0ebe419b4005bec8ba1c8cc82c16bb068e600510a04aaa6e544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-d, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-03T17:41:14.671 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 17:41:14 smithi067 podman[91854]: 2024-05-03 17:41:14.441249087 +0000 UTC m=+4.672392505 container cleanup 3829824985cba0ebe419b4005bec8ba1c8cc82c16bb068e600510a04aaa6e544 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-03T17:41:14.672 INFO:journalctl@ceph.mds.d.smithi067.stdout:May 03 17:41:14 smithi067 bash[91854]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-d 2024-05-03T17:41:14.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.d.service' 2024-05-03T17:41:14.851 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:14.851 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-03T17:41:14.851 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-03T17:41:14.852 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.g 2024-05-03T17:41:15.199 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:41:14 smithi067 systemd[1]: Stopping Ceph mds.g for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:41:15.199 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:41:15 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-g[71989]: 2024-05-03T17:41:15.176+0000 7f4ca8aab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:41:15.200 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:41:15 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-g[71989]: 2024-05-03T17:41:15.176+0000 7f4ca8aab700 -1 mds.g *** got signal Terminated *** 2024-05-03T17:41:19.449 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:41:19 smithi067 podman[91975]: 2024-05-03 17:41:19.190040849 +0000 UTC m=+4.116048657 container died 08c0c8ab3a741232de03883df2e0c41f8c51ded697cdff119cc202506630bdfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-g, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:41:19.924 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:41:19 smithi067 podman[91975]: 2024-05-03 17:41:19.661165055 +0000 UTC m=+4.587172860 container cleanup 08c0c8ab3a741232de03883df2e0c41f8c51ded697cdff119cc202506630bdfc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-g, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux ) 2024-05-03T17:41:19.924 INFO:journalctl@ceph.mds.g.smithi067.stdout:May 03 17:41:19 smithi067 bash[91975]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-g 2024-05-03T17:41:20.089 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.g.service' 2024-05-03T17:41:20.121 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:20.122 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-03T17:41:20.122 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-03T17:41:20.122 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.j 2024-05-03T17:41:20.449 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 17:41:20 smithi067 systemd[1]: Stopping Ceph mds.j for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:41:20.949 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 17:41:20 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-j[70721]: 2024-05-03T17:41:20.455+0000 7f740058e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:41:20.949 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 17:41:20 smithi067 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-j[70721]: 2024-05-03T17:41:20.455+0000 7f740058e700 -1 mds.j *** got signal Terminated *** 2024-05-03T17:41:22.078 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 17:41:21 smithi067 podman[92097]: 2024-05-03 17:41:21.820774062 +0000 UTC m=+1.469669178 container died 3d98710bbba2a98bb63dd7259f825fc0038cc178bcfac72a7feb2e048e6b56d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-j, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-03T17:41:22.449 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 17:41:22 smithi067 podman[92097]: 2024-05-03 17:41:22.15664353 +0000 UTC m=+1.805538646 container cleanup 3d98710bbba2a98bb63dd7259f825fc0038cc178bcfac72a7feb2e048e6b56d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-j, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:41:22.449 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 17:41:22 smithi067 bash[92097]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-j 2024-05-03T17:41:22.449 INFO:journalctl@ceph.mds.j.smithi067.stdout:May 03 17:41:22 smithi067 podman[92111]: 2024-05-03 17:41:22.324201277 +0000 UTC m=+0.500930062 container remove 3d98710bbba2a98bb63dd7259f825fc0038cc178bcfac72a7feb2e048e6b56d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:41:22.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.j.service' 2024-05-03T17:41:22.574 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:22.574 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-03T17:41:22.574 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-03T17:41:22.574 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.b 2024-05-03T17:41:22.853 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:41:22 smithi070 systemd[1]: Stopping Ceph mds.b for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:41:23.223 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:41:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b[63855]: 2024-05-03T17:41:22.855+0000 7efc47818700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:41:23.224 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:41:22 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b[63855]: 2024-05-03T17:41:22.856+0000 7efc47818700 -1 mds.b *** got signal Terminated *** 2024-05-03T17:41:26.462 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:41:26 smithi070 podman[69561]: 2024-05-03 17:41:26.199316813 +0000 UTC m=+3.427334420 container died d4ff0e3af46a6ac44017927f2c1728869f001a04e7a531814fa862f42470cc80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-03T17:41:26.720 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:41:26 smithi070 podman[69561]: 2024-05-03 17:41:26.548605002 +0000 UTC m=+3.776622533 container cleanup d4ff0e3af46a6ac44017927f2c1728869f001a04e7a531814fa862f42470cc80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-03T17:41:26.720 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:41:26 smithi070 bash[69561]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b 2024-05-03T17:41:26.968 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.b.service' 2024-05-03T17:41:26.989 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:41:26 smithi070 podman[69587]: 2024-05-03 17:41:26.724510207 +0000 UTC m=+0.515757529 container remove d4ff0e3af46a6ac44017927f2c1728869f001a04e7a531814fa862f42470cc80 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-05-03T17:41:26.989 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:41:26 smithi070 systemd[1]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.b.service: Deactivated successfully. 2024-05-03T17:41:26.989 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:41:26 smithi070 systemd[1]: Stopped Ceph mds.b for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:41:26.989 INFO:journalctl@ceph.mds.b.smithi070.stdout:May 03 17:41:26 smithi070 systemd[1]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.b.service: Consumed 5.928s CPU time. 2024-05-03T17:41:27.003 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:27.003 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-03T17:41:27.004 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-03T17:41:27.004 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.e 2024-05-03T17:41:27.304 INFO:journalctl@ceph.mds.e.smithi070.stdout:May 03 17:41:27 smithi070 systemd[1]: Stopping Ceph mds.e for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:41:27.804 INFO:journalctl@ceph.mds.e.smithi070.stdout:May 03 17:41:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-e[63223]: 2024-05-03T17:41:27.318+0000 7efddb488700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:41:27.804 INFO:journalctl@ceph.mds.e.smithi070.stdout:May 03 17:41:27 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-e[63223]: 2024-05-03T17:41:27.318+0000 7efddb488700 -1 mds.e *** got signal Terminated *** 2024-05-03T17:41:30.513 INFO:journalctl@ceph.mds.e.smithi070.stdout:May 03 17:41:30 smithi070 podman[69693]: 2024-05-03 17:41:30.196934453 +0000 UTC m=+2.977134733 container died 53db2ce50df772d40502b3f69948d10c78eb5f45a50fdb74e0a5c17e69cca933 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-e, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.license=GPLv2) 2024-05-03T17:41:30.804 INFO:journalctl@ceph.mds.e.smithi070.stdout:May 03 17:41:30 smithi070 podman[69693]: 2024-05-03 17:41:30.517869814 +0000 UTC m=+3.298070096 container cleanup 53db2ce50df772d40502b3f69948d10c78eb5f45a50fdb74e0a5c17e69cca933 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:41:30.804 INFO:journalctl@ceph.mds.e.smithi070.stdout:May 03 17:41:30 smithi070 bash[69693]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-e 2024-05-03T17:41:30.804 INFO:journalctl@ceph.mds.e.smithi070.stdout:May 03 17:41:30 smithi070 podman[69707]: 2024-05-03 17:41:30.71883793 +0000 UTC m=+0.515849072 container remove 53db2ce50df772d40502b3f69948d10c78eb5f45a50fdb74e0a5c17e69cca933 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:41:30.930 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.e.service' 2024-05-03T17:41:30.962 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:30.963 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-03T17:41:30.963 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-03T17:41:30.963 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.h 2024-05-03T17:41:31.284 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:41:31 smithi070 systemd[1]: Stopping Ceph mds.h for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:41:31.554 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:41:31 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-h[63535]: 2024-05-03T17:41:31.287+0000 7f9643397700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:41:31.554 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:41:31 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-h[63535]: 2024-05-03T17:41:31.287+0000 7f9643397700 -1 mds.h *** got signal Terminated *** 2024-05-03T17:41:34.434 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:41:34 smithi070 podman[69811]: 2024-05-03 17:41:34.203655406 +0000 UTC m=+3.021736713 container died 6563c074bc62b64016e0287c086caf3a15f99926be29869918cac9b91a2bd91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-h, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-03T17:41:34.734 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:41:34 smithi070 podman[69811]: 2024-05-03 17:41:34.529674959 +0000 UTC m=+3.347756273 container cleanup 6563c074bc62b64016e0287c086caf3a15f99926be29869918cac9b91a2bd91e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-03T17:41:34.735 INFO:journalctl@ceph.mds.h.smithi070.stdout:May 03 17:41:34 smithi070 bash[69811]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-h 2024-05-03T17:41:34.867 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.h.service' 2024-05-03T17:41:34.898 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:34.898 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-03T17:41:34.898 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-03T17:41:34.898 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.k 2024-05-03T17:41:35.054 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:41:34 smithi070 systemd[1]: Stopping Ceph mds.k for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:41:35.554 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:41:35 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-k[62900]: 2024-05-03T17:41:35.190+0000 7f03a2a08700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:41:35.554 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:41:35 smithi070 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-k[62900]: 2024-05-03T17:41:35.190+0000 7f03a2a08700 -1 mds.k *** got signal Terminated *** 2024-05-03T17:41:37.154 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:41:36 smithi070 podman[69931]: 2024-05-03 17:41:36.829840929 +0000 UTC m=+1.717083071 container died e4257f73457477f32efe8d248ddc571f77a1a6acba192684a1e361a88fc4315e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-k, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:41:37.454 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:41:37 smithi070 podman[69931]: 2024-05-03 17:41:37.257675888 +0000 UTC m=+2.144918039 container cleanup e4257f73457477f32efe8d248ddc571f77a1a6acba192684a1e361a88fc4315e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-k, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:41:37.455 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:41:37 smithi070 bash[69931]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-k 2024-05-03T17:41:37.688 DEBUG:teuthology.orchestra.run.smithi070:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.k.service' 2024-05-03T17:41:37.708 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:41:37 smithi070 podman[69945]: 2024-05-03 17:41:37.458544328 +0000 UTC m=+0.625930924 container remove e4257f73457477f32efe8d248ddc571f77a1a6acba192684a1e361a88fc4315e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-k, CEPH_POINT_RELEASE=, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-05-03T17:41:37.708 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:41:37 smithi070 systemd[1]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.k.service: Deactivated successfully. 2024-05-03T17:41:37.708 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:41:37 smithi070 systemd[1]: Stopped Ceph mds.k for 20c8e22e-0970-11ef-bc97-c7b262605968. 2024-05-03T17:41:37.708 INFO:journalctl@ceph.mds.k.smithi070.stdout:May 03 17:41:37 smithi070 systemd[1]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.k.service: Consumed 1.141s CPU time. 2024-05-03T17:41:37.719 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:37.719 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-03T17:41:37.719 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-03T17:41:37.719 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.c 2024-05-03T17:41:38.010 INFO:journalctl@ceph.mds.c.smithi082.stdout:May 03 17:41:37 smithi082 systemd[1]: Stopping Ceph mds.c for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:41:38.510 INFO:journalctl@ceph.mds.c.smithi082.stdout:May 03 17:41:38 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-c[64699]: 2024-05-03T17:41:38.043+0000 7f095bc4e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:41:38.511 INFO:journalctl@ceph.mds.c.smithi082.stdout:May 03 17:41:38 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-c[64699]: 2024-05-03T17:41:38.043+0000 7f095bc4e700 -1 mds.c *** got signal Terminated *** 2024-05-03T17:41:42.333 INFO:journalctl@ceph.mds.c.smithi082.stdout:May 03 17:41:42 smithi082 podman[70754]: 2024-05-03 17:41:42.057215527 +0000 UTC m=+4.144261410 container died 664901c28883e32ff1597d02f6652610a615de1e89c414245ec8f254a40af606 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.schema-version=1.0) 2024-05-03T17:41:42.649 INFO:journalctl@ceph.mds.c.smithi082.stdout:May 03 17:41:42 smithi082 podman[70754]: 2024-05-03 17:41:42.419098778 +0000 UTC m=+4.506144661 container cleanup 664901c28883e32ff1597d02f6652610a615de1e89c414245ec8f254a40af606 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-03T17:41:42.649 INFO:journalctl@ceph.mds.c.smithi082.stdout:May 03 17:41:42 smithi082 bash[70754]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-c 2024-05-03T17:41:42.789 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.c.service' 2024-05-03T17:41:42.823 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:42.823 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-03T17:41:42.824 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-03T17:41:42.824 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.f 2024-05-03T17:41:43.164 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:41:42 smithi082 systemd[1]: Stopping Ceph mds.f for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:41:43.510 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:41:43 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-f[64389]: 2024-05-03T17:41:43.166+0000 7f478746a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:41:43.511 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:41:43 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-f[64389]: 2024-05-03T17:41:43.166+0000 7f478746a700 -1 mds.f *** got signal Terminated *** 2024-05-03T17:41:46.510 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:41:46 smithi082 podman[70886]: 2024-05-03 17:41:46.210194511 +0000 UTC m=+3.170520190 container died 18fdab0f4e0f5a397c792dc3db7b743d835a4bb8fac54aa66186b6b80dc83f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-f, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-03T17:41:46.836 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:41:46 smithi082 podman[70886]: 2024-05-03 17:41:46.606173693 +0000 UTC m=+3.566499411 container cleanup 18fdab0f4e0f5a397c792dc3db7b743d835a4bb8fac54aa66186b6b80dc83f41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-f, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-05-03T17:41:46.836 INFO:journalctl@ceph.mds.f.smithi082.stdout:May 03 17:41:46 smithi082 bash[70886]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-f 2024-05-03T17:41:46.987 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.f.service' 2024-05-03T17:41:47.019 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:47.019 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-03T17:41:47.019 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-03T17:41:47.020 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.i 2024-05-03T17:41:47.510 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:41:47 smithi082 systemd[1]: Stopping Ceph mds.i for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:41:47.510 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:41:47 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-i[65012]: 2024-05-03T17:41:47.351+0000 7f3bc7361700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:41:47.511 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:41:47 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-i[65012]: 2024-05-03T17:41:47.351+0000 7f3bc7361700 -1 mds.i *** got signal Terminated *** 2024-05-03T17:41:50.481 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:41:50 smithi082 podman[71005]: 2024-05-03 17:41:50.206516235 +0000 UTC m=+2.966551855 container died e05286d474b358251decfe9c59b23e98f794e87906a2a9a940b4a424cd4ea3fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-i, io.buildah.version=1.33.5, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-03T17:41:50.760 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:41:50 smithi082 podman[71005]: 2024-05-03 17:41:50.584973559 +0000 UTC m=+3.345009169 container cleanup e05286d474b358251decfe9c59b23e98f794e87906a2a9a940b4a424cd4ea3fc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-i, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-03T17:41:50.760 INFO:journalctl@ceph.mds.i.smithi082.stdout:May 03 17:41:50 smithi082 bash[71005]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-i 2024-05-03T17:41:51.002 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.i.service' 2024-05-03T17:41:51.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:51.033 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-03T17:41:51.033 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-03T17:41:51.033 DEBUG:teuthology.orchestra.run.smithi082:> sudo systemctl stop ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.l 2024-05-03T17:41:51.510 INFO:journalctl@ceph.mds.l.smithi082.stdout:May 03 17:41:51 smithi082 systemd[1]: Stopping Ceph mds.l for 20c8e22e-0970-11ef-bc97-c7b262605968... 2024-05-03T17:41:51.511 INFO:journalctl@ceph.mds.l.smithi082.stdout:May 03 17:41:51 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-l[64078]: 2024-05-03T17:41:51.337+0000 7f3fa75f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-03T17:41:51.511 INFO:journalctl@ceph.mds.l.smithi082.stdout:May 03 17:41:51 smithi082 ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-l[64078]: 2024-05-03T17:41:51.337+0000 7f3fa75f5700 -1 mds.l *** got signal Terminated *** 2024-05-03T17:41:54.334 INFO:journalctl@ceph.mds.l.smithi082.stdout:May 03 17:41:54 smithi082 podman[71126]: 2024-05-03 17:41:54.05590591 +0000 UTC m=+2.808330087 container died 25e9b16902a57f6be2e0a8a095db4a2e0821590b3dd4bf0b74ed45577684fb9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe) 2024-05-03T17:41:54.676 INFO:journalctl@ceph.mds.l.smithi082.stdout:May 03 17:41:54 smithi082 podman[71126]: 2024-05-03 17:41:54.421502253 +0000 UTC m=+3.173926430 container cleanup 25e9b16902a57f6be2e0a8a095db4a2e0821590b3dd4bf0b74ed45577684fb9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:68e15ae152ef5c805c733e76da8fc1d3dd4ee669d5ba025f457e05725baa24dd, name=ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-l, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-rishabh-testing-20240501.193033-9a047fe, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-03T17:41:54.676 INFO:journalctl@ceph.mds.l.smithi082.stdout:May 03 17:41:54 smithi082 bash[71126]: ceph-20c8e22e-0970-11ef-bc97-c7b262605968-mds-l 2024-05-03T17:41:54.840 DEBUG:teuthology.orchestra.run.smithi082:> sudo pkill -f 'journalctl -f -n 0 -u ceph-20c8e22e-0970-11ef-bc97-c7b262605968@mds.l.service' 2024-05-03T17:41:54.870 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-03T17:41:54.870 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-03T17:41:54.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-03T17:41:55.070 INFO:teuthology.orchestra.run.smithi067.stdout:Deleting cluster with fsid: 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:41:57.547 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-03T17:41:57.735 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:42:00.184 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 --force --keep-logs 2024-05-03T17:42:00.372 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:42:02.642 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T17:42:02.676 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T17:42:02.707 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-03T17:42:02.733 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-03T17:42:02.734 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391/remote/smithi067/crash 2024-05-03T17:42:02.735 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/crash -- . 2024-05-03T17:42:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:tar: /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-03T17:42:02.779 INFO:teuthology.orchestra.run.smithi067.stderr:tar: Error is not recoverable: exiting now 2024-05-03T17:42:02.781 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391/remote/smithi070/crash 2024-05-03T17:42:02.782 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/crash -- . 2024-05-03T17:42:02.818 INFO:teuthology.orchestra.run.smithi070.stderr:tar: /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-03T17:42:02.819 INFO:teuthology.orchestra.run.smithi070.stderr:tar: Error is not recoverable: exiting now 2024-05-03T17:42:02.820 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391/remote/smithi082/crash 2024-05-03T17:42:02.821 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/crash -- . 2024-05-03T17:42:02.861 INFO:teuthology.orchestra.run.smithi082.stderr:tar: /var/lib/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/crash: Cannot open: No such file or directory 2024-05-03T17:42:02.861 INFO:teuthology.orchestra.run.smithi082.stderr:tar: Error is not recoverable: exiting now 2024-05-03T17:42:02.862 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-03T17:42:02.862 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-05-03T17:42:02.898 INFO:tasks.cephadm:Compressing logs... 2024-05-03T17:42:02.899 DEBUG:teuthology.orchestra.run.smithi067:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-03T17:42:02.942 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-03T17:42:02.944 DEBUG:teuthology.orchestra.run.smithi082:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-03T17:42:02.966 INFO:teuthology.orchestra.run.smithi067.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-03T17:42:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:: No such file or directory 2024-05-03T17:42:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.log 2024-05-03T17:42:02.967 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-03T17:42:02.979 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.g.log 2024-05-03T17:42:02.979 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.log: /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.j.log 2024-05-03T17:42:02.979 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.3.log 2024-05-03T17:42:02.979 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.g.log: /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.0.log 2024-05-03T17:42:02.979 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mgr.x.log 2024-05-03T17:42:02.980 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mon.a.log 2024-05-03T17:42:02.980 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mgr.x.log: 89.0% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-03T17:42:02.980 INFO:teuthology.orchestra.run.smithi067.stderr: 90.4% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.audit.log.gzgzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.6.log 2024-05-03T17:42:02.980 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-03T17:42:02.980 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-03T17:42:02.980 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.6.log: 94.5% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.g.log.gz 2024-05-03T17:42:02.980 INFO:teuthology.orchestra.run.smithi067.stderr: 94.7% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.j.log.gz 2024-05-03T17:42:02.980 INFO:teuthology.orchestra.run.smithi067.stderr: 93.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-03T17:42:02.985 INFO:teuthology.orchestra.run.smithi082.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-03T17:42:02.986 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-03T17:42:02.986 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.log 2024-05-03T17:42:02.986 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose --/var/log/ceph/cephadm.log: /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.2.log 2024-05-03T17:42:02.986 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.f.log 2024-05-03T17:42:02.986 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-03T17:42:02.986 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.5.log 2024-05-03T17:42:02.986 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.f.log: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-03T17:42:02.987 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.c.log 2024-05-03T17:42:02.987 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mon.c.log 2024-05-03T17:42:02.987 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mgr.z.log 89.1% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-03T17:42:02.987 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-05-03T17:42:02.987 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.i.log 2024-05-03T17:42:02.987 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-03T17:42:02.987 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.i.log: 90.6% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-03T17:42:02.988 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.11.log 2024-05-03T17:42:02.988 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.cephadm.log: 82.7% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-03T17:42:02.988 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.8.log 2024-05-03T17:42:02.988 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.11.log: 93.2% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mgr.z.log.gz 2024-05-03T17:42:02.991 INFO:teuthology.orchestra.run.smithi082.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-03T17:42:02.993 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.a.log 2024-05-03T17:42:02.993 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-03T17:42:02.993 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.d.log 2024-05-03T17:42:02.993 INFO:teuthology.orchestra.run.smithi067.stderr: 82.9% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-03T17:42:02.993 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-volume.log: 94.6% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.a.log.gz 2024-05-03T17:42:02.994 INFO:teuthology.orchestra.run.smithi070.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-03T17:42:02.995 INFO:teuthology.orchestra.run.smithi070.stderr:: No such file or directory 2024-05-03T17:42:02.995 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.log 2024-05-03T17:42:02.995 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.audit.log 2024-05-03T17:42:02.995 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.7.log 2024-05-03T17:42:02.995 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.b.log 2024-05-03T17:42:02.995 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.4.log 2024-05-03T17:42:02.995 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.7.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-03T17:42:02.995 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.1.log 2024-05-03T17:42:02.995 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mgr.y.log 2024-05-03T17:42:02.996 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.h.log 2024-05-03T17:42:02.996 INFO:teuthology.orchestra.run.smithi070.stderr: 89.1%/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mgr.y.log: -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.log.gz 2024-05-03T17:42:02.996 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.10.log 2024-05-03T17:42:02.996 INFO:teuthology.orchestra.run.smithi070.stderr: 90.6% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.audit.log.gz 2024-05-03T17:42:02.996 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.cephadm.log 2024-05-03T17:42:02.996 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.h.log: /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.e.log 2024-05-03T17:42:02.996 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.cephadm.log: 82.8% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph.cephadm.log.gz 2024-05-03T17:42:02.996 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-volume.log 2024-05-03T17:42:02.996 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.e.log: 93.1% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mgr.y.log.gz 2024-05-03T17:42:02.997 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mon.b.log 2024-05-03T17:42:03.000 INFO:teuthology.orchestra.run.smithi067.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.9.log 2024-05-03T17:42:03.001 INFO:teuthology.orchestra.run.smithi067.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.d.log: /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.9.log: 91.1% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-03T17:42:03.002 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-volume.log: 91.0% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-03T17:42:03.004 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.l.log 2024-05-03T17:42:03.005 INFO:teuthology.orchestra.run.smithi067.stderr: 94.6% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.d.log.gz 2024-05-03T17:42:03.006 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.k.log 2024-05-03T17:42:03.029 INFO:teuthology.orchestra.run.smithi082.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-volume.log: /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.l.log: 94.1% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.i.log.gz 2024-05-03T17:42:03.031 INFO:teuthology.orchestra.run.smithi082.stderr: 91.0% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-volume.log.gz 2024-05-03T17:42:03.031 INFO:teuthology.orchestra.run.smithi082.stderr: 94.7% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.l.log.gz 2024-05-03T17:42:03.035 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mon.b.log: /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.k.log: 94.4% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.k.log.gz 2024-05-03T17:42:03.035 INFO:teuthology.orchestra.run.smithi070.stderr: 94.0% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.h.log.gz 2024-05-03T17:42:03.049 INFO:teuthology.orchestra.run.smithi070.stderr: 94.6% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.e.log.gz 2024-05-03T17:42:03.051 INFO:teuthology.orchestra.run.smithi082.stderr: 94.6% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.f.log.gz 2024-05-03T17:42:03.057 INFO:teuthology.orchestra.run.smithi082.stderr: 94.5% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.c.log.gz 2024-05-03T17:42:03.177 INFO:teuthology.orchestra.run.smithi070.stderr: 93.6% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mds.b.log.gz 2024-05-03T17:42:03.289 INFO:teuthology.orchestra.run.smithi082.stderr: 92.2% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mon.c.log.gz 2024-05-03T17:42:03.319 INFO:teuthology.orchestra.run.smithi067.stderr: 90.5% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mgr.x.log.gz 2024-05-03T17:42:03.441 INFO:teuthology.orchestra.run.smithi070.stderr: 92.2% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mon.b.log.gz 2024-05-03T17:42:03.577 INFO:teuthology.orchestra.run.smithi082.stderr: 92.9% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.11.log.gz 2024-05-03T17:42:03.594 INFO:teuthology.orchestra.run.smithi082.stderr: 92.6% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.8.log.gz 2024-05-03T17:42:03.627 INFO:teuthology.orchestra.run.smithi082.stderr: 92.5% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.2.log.gz 2024-05-03T17:42:03.636 INFO:teuthology.orchestra.run.smithi070.stderr: 92.6% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.7.log.gz 2024-05-03T17:42:03.669 INFO:teuthology.orchestra.run.smithi070.stderr: 92.4% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.1.log.gz 2024-05-03T17:42:03.706 INFO:teuthology.orchestra.run.smithi070.stderr: 92.3% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.4.log.gz 2024-05-03T17:42:03.892 INFO:teuthology.orchestra.run.smithi082.stderr: 92.3% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.5.log.gz 2024-05-03T17:42:03.893 INFO:teuthology.orchestra.run.smithi082.stderr: 2024-05-03T17:42:03.893 INFO:teuthology.orchestra.run.smithi082.stderr:real 0m0.935s 2024-05-03T17:42:03.894 INFO:teuthology.orchestra.run.smithi082.stderr:user 0m3.235s 2024-05-03T17:42:03.894 INFO:teuthology.orchestra.run.smithi082.stderr:sys 0m0.135s 2024-05-03T17:42:04.006 INFO:teuthology.orchestra.run.smithi067.stderr: 93.1% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.9.log.gz 2024-05-03T17:42:04.041 INFO:teuthology.orchestra.run.smithi067.stderr: 90.4% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-mon.a.log.gz 2024-05-03T17:42:04.119 INFO:teuthology.orchestra.run.smithi067.stderr: 93.1% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.3.log.gz 2024-05-03T17:42:04.406 INFO:teuthology.orchestra.run.smithi067.stderr: 93.1% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.0.log.gz 2024-05-03T17:42:04.448 INFO:teuthology.orchestra.run.smithi070.stderr: 92.8% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.10.log.gz 2024-05-03T17:42:04.451 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-05-03T17:42:04.451 INFO:teuthology.orchestra.run.smithi070.stderr:real 0m1.493s 2024-05-03T17:42:04.451 INFO:teuthology.orchestra.run.smithi070.stderr:user 0m4.243s 2024-05-03T17:42:04.451 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m0.167s 2024-05-03T17:42:04.503 INFO:teuthology.orchestra.run.smithi067.stderr: 92.8% -- replaced with /var/log/ceph/20c8e22e-0970-11ef-bc97-c7b262605968/ceph-osd.6.log.gz 2024-05-03T17:42:04.505 INFO:teuthology.orchestra.run.smithi067.stderr: 2024-05-03T17:42:04.505 INFO:teuthology.orchestra.run.smithi067.stderr:real 0m1.549s 2024-05-03T17:42:04.505 INFO:teuthology.orchestra.run.smithi067.stderr:user 0m6.396s 2024-05-03T17:42:04.505 INFO:teuthology.orchestra.run.smithi067.stderr:sys 0m0.223s 2024-05-03T17:42:04.505 INFO:tasks.cephadm:Archiving logs... 2024-05-03T17:42:04.506 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391/remote/smithi067/log 2024-05-03T17:42:04.507 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-03T17:42:05.422 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391/remote/smithi070/log 2024-05-03T17:42:05.423 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-03T17:42:06.071 DEBUG:teuthology.misc:Transferring archived files from smithi082:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391/remote/smithi082/log 2024-05-03T17:42:06.072 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-03T17:42:06.597 INFO:tasks.cephadm:Removing cluster... 2024-05-03T17:42:06.597 DEBUG:teuthology.orchestra.run.smithi067:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 --force 2024-05-03T17:42:06.801 INFO:teuthology.orchestra.run.smithi067.stdout:Deleting cluster with fsid: 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:42:07.071 DEBUG:teuthology.orchestra.run.smithi070:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 --force 2024-05-03T17:42:07.261 INFO:teuthology.orchestra.run.smithi070.stdout:Deleting cluster with fsid: 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:42:07.520 DEBUG:teuthology.orchestra.run.smithi082:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 20c8e22e-0970-11ef-bc97-c7b262605968 --force 2024-05-03T17:42:07.701 INFO:teuthology.orchestra.run.smithi082.stdout:Deleting cluster with fsid: 20c8e22e-0970-11ef-bc97-c7b262605968 2024-05-03T17:42:07.958 INFO:tasks.cephadm:Removing cephadm ... 2024-05-03T17:42:07.959 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-03T17:42:07.977 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-03T17:42:07.992 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-03T17:42:08.007 INFO:tasks.cephadm:Teardown complete 2024-05-03T17:42:08.007 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-03T17:42:08.017 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-03T17:42:08.017 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-03T17:42:08.020 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-03T17:42:08.035 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-03T17:42:08.143 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-03T17:42:08.152 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-03T17:42:08.152 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T17:42:08.155 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T17:42:08.157 DEBUG:teuthology.orchestra.run.smithi082:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-03T17:42:08.169 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-05-03T17:42:08.169 INFO:teuthology.orchestra.run.smithi067.stderr:bash: line 1: ntpq: command not found 2024-05-03T17:42:08.170 INFO:teuthology.orchestra.run.smithi082.stderr:bash: line 1: ntpq: command not found 2024-05-03T17:42:08.172 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T17:42:08.173 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-05-03T17:42:08.173 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv01.front.sepia.ceph.com 3 6 377 35 +700us[ +658us] +/- 52ms 2024-05-03T17:42:08.173 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 36 -345us[ -387us] +/- 66ms 2024-05-03T17:42:08.173 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 -609us[ -651us] +/- 68ms 2024-05-03T17:42:08.173 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:42:08.174 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T17:42:08.175 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2024-05-03T17:42:08.175 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv01.front.sepia.ceph.com 3 6 377 35 +683us[ +681us] +/- 52ms 2024-05-03T17:42:08.175 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 31 -384us[ -384us] +/- 66ms 2024-05-03T17:42:08.175 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 -634us[ -636us] +/- 68ms 2024-05-03T17:42:08.175 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:42:08.176 INFO:teuthology.orchestra.run.smithi082.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-03T17:42:08.176 INFO:teuthology.orchestra.run.smithi082.stdout:=============================================================================== 2024-05-03T17:42:08.176 INFO:teuthology.orchestra.run.smithi082.stdout:^* hv01.front.sepia.ceph.com 3 6 377 36 +708us[ +708us] +/- 52ms 2024-05-03T17:42:08.176 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -347us[ -347us] +/- 66ms 2024-05-03T17:42:08.177 INFO:teuthology.orchestra.run.smithi082.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 -590us[ -590us] +/- 68ms 2024-05-03T17:42:08.177 INFO:teuthology.orchestra.run.smithi082.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-03T17:42:08.177 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-03T17:42:08.188 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-03T17:42:08.188 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-03T17:42:08.202 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-03T17:42:08.294 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-03T17:42:08.357 DEBUG:teuthology.orchestra.run.smithi082:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-03T17:42:08.422 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-03T17:42:08.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:42:08.452 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-03T17:42:08.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:42:08.479 DEBUG:teuthology.orchestra.run.smithi082:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-03T17:42:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:42:08.505 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-03T17:42:08.515 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-03T17:42:08.530 INFO:teuthology.task.internal:Duration was 2233.677077 seconds 2024-05-03T17:42:08.530 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-03T17:42:08.545 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-03T17:42:08.546 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-03T17:42:08.548 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-03T17:42:08.551 DEBUG:teuthology.orchestra.run.smithi082:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-03T17:42:08.585 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T17:42:08.586 INFO:teuthology.orchestra.run.smithi082.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T17:42:08.588 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-03T17:42:09.096 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-03T17:42:09.096 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2024-05-03T17:42:09.097 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-03T17:42:09.121 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-05-03T17:42:09.121 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-03T17:42:09.160 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi082.front.sepia.ceph.com 2024-05-03T17:42:09.160 DEBUG:teuthology.orchestra.run.smithi082:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-03T17:42:09.188 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-03T17:42:09.189 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-03T17:42:09.191 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-03T17:42:09.203 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-03T17:42:09.254 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-03T17:42:09.254 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-03T17:42:09.262 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-03T17:42:09.266 DEBUG:teuthology.orchestra.run.smithi082:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-03T17:42:09.759 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-03T17:42:09.769 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-03T17:42:09.769 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-03T17:42:09.794 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-03T17:42:09.819 DEBUG:teuthology.orchestra.run.smithi082:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-03T17:42:09.844 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-03T17:42:09.856 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-03T17:42:09.858 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-03T17:42:09.861 DEBUG:teuthology.orchestra.run.smithi082:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-03T17:42:09.878 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2024-05-03T17:42:09.881 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-05-03T17:42:09.907 INFO:teuthology.orchestra.run.smithi082.stdout:kernel.core_pattern = core 2024-05-03T17:42:09.931 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-03T17:42:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:42:09.946 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-03T17:42:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:42:09.960 DEBUG:teuthology.orchestra.run.smithi082:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-03T17:42:09.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-03T17:42:09.985 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-03T17:42:09.995 INFO:teuthology.task.internal:Transferring archived files... 2024-05-03T17:42:09.996 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391/remote/smithi067 2024-05-03T17:42:09.996 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-03T17:42:10.041 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391/remote/smithi070 2024-05-03T17:42:10.041 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-03T17:42:10.075 DEBUG:teuthology.misc:Transferring archived files from smithi082:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391/remote/smithi082 2024-05-03T17:42:10.076 DEBUG:teuthology.orchestra.run.smithi082:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-03T17:42:10.109 INFO:teuthology.task.internal:Removing archive directory... 2024-05-03T17:42:10.109 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-03T17:42:10.112 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-03T17:42:10.114 DEBUG:teuthology.orchestra.run.smithi082:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-03T17:42:10.158 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-03T17:42:10.171 INFO:teuthology.task.internal:Not uploading archives. 2024-05-03T17:42:10.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-03T17:42:10.183 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-03T17:42:10.183 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-03T17:42:10.186 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-03T17:42:10.188 DEBUG:teuthology.orchestra.run.smithi082:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-03T17:42:10.198 INFO:teuthology.orchestra.run.smithi067.stdout: 262213 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 3 17:42 /home/ubuntu/cephtest 2024-05-03T17:42:10.200 INFO:teuthology.orchestra.run.smithi070.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 3 17:42 /home/ubuntu/cephtest 2024-05-03T17:42:10.213 INFO:teuthology.orchestra.run.smithi082.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 3 17:42 /home/ubuntu/cephtest 2024-05-03T17:42:10.214 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-03T17:42:10.226 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-03T17:42:10.247 INFO:teuthology.nuke:Checking targets against current locks 2024-05-03T17:42:10.274 DEBUG:teuthology.nuke:shortname: smithi067 2024-05-03T17:42:10.274 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-03T17:42:10.299 DEBUG:teuthology.nuke:shortname: smithi070 2024-05-03T17:42:10.299 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-03T17:42:10.305 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 16:59:12.192530', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEghdIU/zEWluAfDO+6rnEN0WLyvfO5LaMhof4BZIY/yquAmAhE0/8SNhwgf3fyCuI2mEOlh5LoX95aGqe/YkSk='} 2024-05-03T17:42:10.321 DEBUG:teuthology.nuke:shortname: smithi082 2024-05-03T17:42:10.321 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-03T17:42:10.325 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 16:59:12.190566', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAF1kzR9TV75MPp4yQ3LcDRoCrShYqaoDFQ2qELokT0YodJ0JFitAOHRwSMtv4yiFyuR44T4XAtpVvS5Mqr6DQY='} 2024-05-03T17:42:10.346 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi082.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-05-03_13:56:19-fs-wip-rishabh-testing-20240501.193033-testing-default-smithi/7687391', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-03 16:59:12.194261', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBHV1K9B4ZJO3ai6nVSN2vyZxcXSDIAQmpRbv8PH+eQYACgkt58P1vYnFQVfylu2nF4WQuzmAas9/WFPXLkdRCA='} 2024-05-03T17:42:10.356 INFO:teuthology.orchestra.console.smithi067:Power off 2024-05-03T17:42:10.356 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-03T17:42:10.382 DEBUG:teuthology.orchestra.console.smithi067:power off output: Chassis Power Control: Down/Off 2024-05-03T17:42:10.382 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T17:42:10.397 INFO:teuthology.orchestra.console.smithi070:Power off 2024-05-03T17:42:10.398 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-03T17:42:10.410 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-05-03T17:42:10.419 DEBUG:teuthology.orchestra.console.smithi070:power off output: Chassis Power Control: Down/Off 2024-05-03T17:42:10.420 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T17:42:10.437 INFO:teuthology.orchestra.console.smithi082:Power off 2024-05-03T17:42:10.438 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-03T17:42:10.447 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-05-03T17:42:10.459 DEBUG:teuthology.orchestra.console.smithi082:power off output: Chassis Power Control: Down/Off 2024-05-03T17:42:10.460 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T17:42:10.483 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-05-03T17:42:14.411 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T17:42:14.448 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T17:42:14.484 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T17:42:14.522 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is on 2024-05-03T17:42:14.559 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-05-03T17:42:14.593 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is on 2024-05-03T17:42:18.527 DEBUG:teuthology.orchestra.console.smithi067:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T17:42:18.560 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T17:42:18.594 DEBUG:teuthology.orchestra.console.smithi082:pexpect command: ipmitool -H smithi082.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-03T17:42:18.636 DEBUG:teuthology.orchestra.console.smithi067:check power output: Chassis Power is off 2024-05-03T17:42:18.670 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is off 2024-05-03T17:42:18.704 DEBUG:teuthology.orchestra.console.smithi082:check power output: Chassis Power is off 2024-05-03T17:42:18.737 INFO:teuthology.orchestra.console.smithi067:Power off completed 2024-05-03T17:42:18.771 INFO:teuthology.orchestra.console.smithi070:Power off completed 2024-05-03T17:42:18.804 INFO:teuthology.orchestra.console.smithi082:Power off completed 2024-05-03T17:42:18.911 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/yes 5-quiesce/with-quiesce 6-workunit/suites/dbench}} duration: 2233.6770765781403 failure_reason: 'Command failed on smithi067 with status 110: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs qa random 0.10''"' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f0bf7852e935487296eb382aca021d20 status: fail success: false 2024-05-03T17:42:18.911 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-03T17:42:19.005 INFO:teuthology.run:FAIL